GRPO, SFT and LoRA: How a Model Learns From Scores

chapter 1

The question in one picture

A writing model gets one plot and a target author. It writes 16 stories for that one prompt, and a scorer rates each one. Story style §5 The sketch shows this step with four stories and made-up scores.

Each story is then compared with its own group. Stories above the group average become more likely, and stories below it become less likely. DeepSeekMath §4.1.2 The DeepSeekMath paper names this method GRPO. DeepSeekMath §4.1.1

The page draws on three papers. DeepSeekMath introduced GRPO. DeepSeekMath §1 The second introduces LoRA, a way to train a big model by changing only a small add-on. LoRA Abstract The third uses GRPO for story style. Story style Abstract

The page follows one question: how does a score, in place of an example answer, change a model?

chapter 2

What you need first

1. A model writes one token at a time

A language model writes text one token at a time. A token is a word or a piece of a word. For each next token, the model gives every option a probability. Training changes the model so that some tokens become more likely. DeepSeekMath A.1.1 This page calls that a push: a token pushed up becomes more likely, and one pushed down becomes less likely. The papers call the model being trained the policy model. DeepSeekMath §4.1.1

2. SFT copies example answers

SFT trains a model on answers that people chose. DeepSeekMath §5.2.1 Every token of every example is pushed up by the same amount. DeepSeekMath A.1.1 No answer is ever pushed down.

The story paper also starts with SFT. Its examples come from another model: 80K long stories written by GPT-oss-120B. Story style §4

3. A score can replace the example

Reinforcement learning needs no example answer. The model writes its own answers, and something scores them. DeepSeekMath §5.2.1 In DeepSeekMath the scorer is a reward model: a second model trained to score answers. DeepSeekMath §5.2.1 The number it gives is the reward. DeepSeekMath §4.1.2

In the story paper, the main scorer compares a new story with a passage by the target author. It is called a style-similarity judge. Story style Abstract

chapter 3

How the training works

One formula for many methods

The DeepSeekMath paper writes SFT, GRPO and several other methods as one formula. It calls this a unified paradigm. DeepSeekMath §5.2.1 In every method, training pushes up the probability of each token in an answer. Each push is multiplied by a number, the gradient coefficient. DeepSeekMath §5.2.1

The methods differ in where the answers come from. They also differ in the size and sign of that number. DeepSeekMath §5.2.1

RFT and Online RFT push right answers up and ignore wrong ones. DeepSeekMath A.1.2 Of the four, only GRPO sets the push from the score, so only it can push a bad answer down. DeepSeekMath §5.2.1

Online RFT and GRPO also use online sampling: the answers come from the model as it is during training. RFT uses answers from the starting model only. DeepSeekMath §5.2.1

From PPO to GRPO

GRPO changes an older method, PPO. DeepSeekMath §4.1.1 PPO trains a second model next to the policy model, called the value model. It guesses the score an answer should get. That guess is the baseline: an answer that beats it counts as good. DeepSeekMath §4.1.1

Figure 4 from the DeepSeekMath paper: PPO (top) and GRPO (bottom) DeepSeekMath Figure 4 open full size

PPO trains a value model next to the policy model (both yellow). Its output feeds the advantage. GRPO drops it.

GRPO takes one question q and samples a group of G outputs, o1 to oG.

The reward model scores each output. A group step turns the scores r1 to rG into advantages A1 to AG.

The reference model stays frozen. In GRPO, the KL divergence to it is a separate penalty in training.

GRPO drops the value model. The average reward of a group of answers to the same question becomes the baseline. DeepSeekMath §4.1.1

GRPO step by step

1. Take one question. Sample a group of answers from the policy model. DeepSeekMath §4.1.2
2. Score each answer with the reward model. DeepSeekMath §4.1.2
3. Subtract the group average from each score. Divide by the spread of the group's scores. The result is the advantage. DeepSeekMath §4.1.2
4. Every token in an answer gets that answer's advantage. Tokens with a positive advantage are pushed up, and tokens with a negative one are pushed down. DeepSeekMath A.1.6

Two parts keep each update small. Clipping limits how far one update can move a token's probability. DeepSeekMath §4.1.1 The KL divergence measures how far the model has drifted from a frozen copy, the reference model. That copy is usually the SFT model. DeepSeekMath §4.1.1 The KL coefficient sets how much this drift costs. DeepSeekMath §4.1.1

LoRA: train a small add-on

Full fine-tuning changes every weight in the model, so each trained copy is as large as the original. LoRA §1

LoRA keeps those weights frozen. For a chosen weight matrix W, it trains two thin tables, B and A. Their product BA is added to W. LoRA §4.1

Figure 1 from the LoRA paper LoRA §4.1 open full size

The pretrained weight matrix W stays frozen. Training does not change it.

Two thin tables are trained instead. A starts random. B starts at zero, so at first nothing changes.

r is the rank: the thin side of A and B. It can be far smaller than d, the size of the input x.

The input x goes through both paths. The two results are added to give h.

The thin side of B and A is the rank, r. LoRA §4.1 B starts at zero, so training starts from the unchanged model. LoRA §4.1

On GPT-3 175B, LoRA cut the memory needed for training from 1.2TB to 350GB. LoRA §4.2

LoRA and GRPO answer different questions. GRPO decides how hard to push each token. LoRA decides which numbers that push may change. The LoRA paper tests its method only on tasks with labelled answers. It does not test reinforcement learning. LoRA §5

The writing system: a judge, then SFT, then GRPO

The story paper builds its style judge first. It is a sentence-transformer trained on pairs of book passages. Some pairs share an author, and some do not. Story style §3.2 The judge also filters the SFT stories before training. Story style §4

The reward mixes three scores. A style reward comes from the judge. Another model gives a content reward for story quality. A completeness reward checks length and the final "THE END" line. Story style §5 Style counts most: 0.6 of the total. Content counts 0.3 and completeness 0.1. Story style §5

chapter 4

What they found

GRPO raised DeepSeekMath's scores after SFT

GRPO started from DeepSeekMath-Instruct 7B, the model after SFT. DeepSeekMath §4.2 On GSM8K, accuracy rose from 82.9% to 88.2%. On MATH, it rose from 46.8% to 51.7%. DeepSeekMath §1

A smaller study compared the methods from the table. On two math tests, Online RFT beat RFT late in training, and GRPO beat Online RFT. DeepSeekMath §5.2.1 DeepSeekMath Figure 5

What reinforcement learning changed, in the authors' reading

Maj@K samples K answers and keeps the most common one. Pass@K counts a question as answered correctly if any of the K answers is right. The paper does not define them; the glossary gives the usual meaning.

Figure 7 from the DeepSeekMath paper DeepSeekMath Figure 7 open full size

At K = 1, one answer per question, the model after reinforcement learning scores higher than before.

Maj@K: orange is after training, purple is before. Orange stays above purple at every K.

Pass@K: the blue and green lines end close together. On GSM8K, the model before reinforcement learning ends higher.

At K = 1 the two measures are the same single answer, and it improved. As K grows, Maj@K stayed higher after reinforcement learning, but Pass@K did not. DeepSeekMath §5.2.2 The authors write: "it seems that the improvement is attributed to boosting the correct response from TopK". DeepSeekMath §5.2.2

In plain words, the right answer was often already among the samples, and reinforcement learning made it come up more often. "It seems" marks this as the authors' reading of the curves.

LoRA matched full fine-tuning

On GPT-3 175B, LoRA matched or beat full fine-tuning on all three test sets. LoRA §5.5 A very small rank was enough. With a rank of 1, the WikiSQL test scored 73.4%. A rank of 64 scored 73.5%. LoRA Table 6

The story models beat larger models on the judge's score

The paper reports a style score: the judge's number, stretched so that 0 looks like a different author and 1 like the same author. Story style §5

The four GRPO models reached style scores from 0.807 to 0.984. Story style Table 3 The average was 0.893. Story style Abstract The best comparison model, GPT-oss-120B, scored 0.370 to 0.663. Story style Table 3 It is far larger than the 8B story model. Story style Abstract

chapter 5

What this does not show

These limits come from the papers, except in the boxes marked otherwise.

• Reinforcement learning may not add new ability. Pass@K did not rise at larger K. The authors link this to training on SFT questions only, with a plain way of sampling answers. DeepSeekMath §5.2.3
• Scores can be wrong. Every method here trusts its reward fully. Even one carefully labelled set of math steps has about 20% wrong labels. DeepSeekMath §5.2.3
• GRPO was tested on math. The DeepSeekMath results come from math tests only. DeepSeekMath §4.2
• LoRA covered only some weights. The paper adds it to some weight tables in each layer and leaves the rest to future work. LoRA §4.2 Mixing tasks with different add-ons in one batch is also hard once they are merged into W. LoRA §4.2
• The story study is small. It uses 50 plots and one prompt format. Story style Limitations The judge may also reward shared words and topics. Story style §5

chapter 6

Words to know

Every term on this page, in the order it appears. Each entry names the paper its definition comes from.

policy model (model in training)

The model being trained. It is the one that writes the answers.

"where π_θ and π_θ_old are the current and old policy models" (DeepSeekMath §4.1.1)

reward (score)

The number an answer gets from the scorer. Higher is better.

"A reward model is then used to score the outputs, yielding G rewards r={r_1, r_2, ..., r_G} correspondingly." (DeepSeekMath §4.1.2)

GRPO (group-score training)

The model writes several answers to one question. Each answer is judged against the group's average score.

"we propose Group Relative Policy Optimization (GRPO), which obviates the need for additional value function approximation as in PPO, and instead uses the average reward of multiple sampled outputs, produced in response to the same question, as the baseline" (DeepSeekMath §4.1.1)

LoRA (the small add-on update)

Keep the model's weights fixed. Train two thin tables whose product is added to each chosen weight table.

"We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks." (LoRA Abstract)

SFT (training on examples)

Training a model to copy good example answers, one token at a time.

"Supervised Fine-tuning (SFT): SFT fine-tunes pretrained model on human selected SFT data." (DeepSeekMath §5.2.1)

reinforcement learning

Training a model by letting it try, scoring what it made, and making high-scoring work more likely.

"Reinforcement learning (RL) has been proven to be effective in further improving the mathematical reasoning ability of LLMs after the Supervised Fine-Tuning (SFT) stage" (DeepSeekMath §4.1)

reward model (scoring model)

A separate model that gives each answer a score.

"Model denotes that we train a reward model to score each response." (DeepSeekMath §5.2.1)

style-similarity judge (style scorer)

A model that reads two texts and gives a number for how alike their writing style is.

"we train a dedicated style-similarity judge by fine-tuning a sentence-transformer with authorship-verification supervision, and calibrate its similarity scores into a bounded [0,1] reward" (Story style Abstract)

unified paradigm (one shared formula)

Writing many training methods as the same formula, so you can see which part differs.

"we provide a unified paradigm to analyze different training methods, such as SFT, RFT, DPO, PPO, GRPO" (DeepSeekMath §5.2.1)

gradient coefficient (push size)

A number for each token: how hard training pushes that token up (positive) or down (negative).

"Algorithm A: which processes the training data and the reward signal to the gradient coefficient GC that determines the magnitude of the penalty or reinforcement for the data." (DeepSeekMath §5.2.1)

advantage (score above the group average)

How much better or worse one answer scored than the others in its group. It can be below zero.

"Â_i,t is the advantage calculated based on relative rewards of the outputs inside each group only" (DeepSeekMath §4.1.1)

online sampling (answers from the current model)

Training answers are written by the model as it is now, not by the model it started from.

"Online sampling denotes that the training data is from the exploration results of the real-time training policy model, while offline sampling denotes that the training data is from the sampling results of the initial SFT model." (DeepSeekMath §5.2.1)

PPO (the value-model method)

The common reinforcement learning method before GRPO. It trains a second model to judge each token.

"Proximal Policy Optimization (PPO) is an actor-critic RL algorithm that is widely used in the RL fine-tuning stage of LLMs" (DeepSeekMath §4.1.1)

value model (second judging model)

A second model, about as big as the one in training, that guesses how good each point in an answer is.

"As the value function employed in PPO is typically another model of comparable size as the policy model, it brings a substantial memory and computational burden." (DeepSeekMath §4.1.1)

also called in the paper: value function, critic model

reference model (frozen starting copy)

A copy of the model from before this training. It does not change.

"π_ref is the reference model, which is usually the initial SFT model" (DeepSeekMath §4.1.1)

KL divergence (distance from the starting copy)

A number that grows as the trained model's word choices drift away from the starting copy's.

"GRPO regularizes by directly adding the KL divergence between the trained policy and the reference policy to the loss" (DeepSeekMath §4.1.1)

clipping (a cap on each change)

A limit on how far one update can move the probability of a token.

"ε is a clipping-related hyper-parameter introduced in PPO for stabilizing training." (DeepSeekMath §4.1.1)

KL coefficient (drift penalty weight)

How strongly training is held back from drifting away from the starting copy. The papers call it beta.

"β is the coefficient of the KL penalty" (DeepSeekMath §4.1.1; Story style §5)

outcome supervision (one score per answer)

The whole answer gets one score at the end.

"Outcome supervision only provides a reward at the end of each output" (DeepSeekMath §4.1.3)

process supervision (a score for each step)

Each step of the answer gets its own score.

"we also explore process supervision, which provides a reward at the end of each reasoning step" (DeepSeekMath §4.1.3)

full fine-tuning (training every weight)

Further training that changes every weight in the model.

"As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible." (LoRA Abstract)

weight matrix (grid of weights)

A large table of numbers inside the model. Text passes through many of them.

"A neural network contains many dense layers which perform matrix multiplication. The weight matrices in these layers typically have full-rank." (LoRA §4.1)

rank (add-on width)

The width r of the two thin tables. A small rank means few numbers to train.

"we constrain its update by representing the latter with a low-rank decomposition W_0+ΔW=W_0+BA, where B∈R^(d×r), A∈R^(r×k), and the rank r ≪ min(d,k)" (LoRA §4.1)

inference latency (extra wait per answer)

How long the model takes to produce output once it is trained.

"Critically, this guarantees that we do not introduce any additional latency during inference compared to a fine-tuned model by construction." (LoRA §4.1)

sentence-transformer (text-comparing model)

A model that turns a text into a list of numbers, so two texts can be compared.

"we explore a sentence-transformer baseline that uses pretrained embeddings to compute pairwise similarity between text chunks" (Story style §3.1)

style reward (style score during training)

The style judge's raw number for a new story, used as most of the reward.

"the fine-tuned sentence-transformer judge produces a raw similarity score S_sim between each generated story and its in-batch reference, and we use this score directly as the GRPO style reward" (Story style §5)

content reward (story-quality score)

Another model grades the story from 1 to 9 on a fixed rubric.

"We evaluate narrative quality, including coherence, characterisation, and readability, using openchat-3.5-0106." (Story style §5)

completeness reward (length-and-ending score)

Full marks only if the story is the right length and ends with THE END.

"This reward checks whether the story meets the length requirement and ends with the required marker." (Story style §5)

Maj@K (majority-vote accuracy)

Sample K answers and take the most common final answer. Count how often that answer is right.

not defined in the paper

Pass@K (any-of-K accuracy)

Sample K answers. Count a question as solved if any one of them is right.

not defined in the paper

style score (rescaled style number)

The judge's number, stretched so that 0 looks like a different author and 1 looks like the same author.

"For each generated story, the style judge outputs a raw similarity s, which we calibrate using benchmark quartiles and clamp to [0,1]." (Story style §5)

GPT-oss-120B

An open model from OpenAI with 120 billion weights. It wrote the SFT stories and is the strongest comparison model.

Story style §4

RFT (training on its own right answers)

Let the model answer, throw away the wrong answers, and train on the right ones.

"RFT further fine-tunes the SFT model on the filtered outputs sampled from the SFT model based on SFT questions. RFT filters the outputs based on the correctness of their answers." (DeepSeekMath §5.2.1)

Online RFT (training on its latest right answers)

Like RFT, but the answers come from the model as it is right now, during training.

"Different from RFT, Online RFT initiates the policy model using the SFT model and refines it by fine-tuning with the augmented outputs sampled from the real-time policy model." (DeepSeekMath §5.2.1)

GPT-3 175B

A language model from OpenAI with 175 billion weights. The largest model the LoRA paper tests.

LoRA Abstract

DrGRPO

A changed version of the GRPO loss from another paper. The story paper uses it.

Story style §5

DeepSeekMath-Instruct 7B

The DeepSeekMath model after SFT and before GRPO. It has 7 billion weights.

DeepSeekMath §3

GSM8K

A test set of grade-school math word problems.

DeepSeekMath §1.2

MATH

A test set of competition math problems, harder than GSM8K.

DeepSeekMath §1.2

CMATH

A Chinese math test set. GRPO never trained on it.

DeepSeekMath §1

WikiSQL

A test set where a model turns an English question into a database query.

LoRA §5