Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11GRPO removes the learned value function, or critic, that PPO normally uses to estimate a baseline. It does not remove the step that scores outputs. A GRPO run still needs a reward for each sampled completion, and that reward can come from a learned reward model, a custom reward function, or another task-specific scorer. The critic and the reward mechanism do different jobs, so removing one does not logically remove the other.
Two components that are easy to conflate
PPO-style reinforcement learning for language models is often described with several moving parts. Two of them matter for this question:
As an Amazon Associate I earn from qualifying purchases.
- The critic (value function) estimates how good a state is, which PPO uses as a baseline when computing advantages. It is a learned model that is trained alongside the policy.
- The reward mechanism assigns a score to each completion the policy produces. It answers “how good was this output?” rather than “what did we expect to get here?”
GRPO (Group Relative Policy Optimization) replaces the first with a statistic computed from the group of outputs. It leaves the second in place. That is the whole substitution, and it explains why the headline’s distinction holds. GRPO was introduced by the DeepSeekMath authors as a variant of PPO, and the paper’s abstract describes it this way:
Free tools Windows power users keep installed
One-click scans. No signup required.
“Second, we introduce Group Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization (PPO), that enhances mathematical reasoning abilities while concurrently optimizing the memory usage of PPO.” (DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, arXiv:2402.03300, 2024)
#1 Best Overall
PPO and GRPO side by side
| Aspect | PPO | GRPO |
|---|---|---|
| Baseline for advantages | Learned critic (value function) | Group-relative statistics: each reward compared with the mean and standard deviation of rewards for the same prompt |
| Source of reward scores | Set by the implementation; can be a learned reward model, a rule, or a custom function | Set by the implementation; can be a learned reward model, a rule, or a custom function |
| Completions per prompt | Not stated in the cited sources | Multiple completions sampled for each prompt |
| Separate value model to train | Yes | No, in the documented formulation |
| KL penalty to a reference policy | Implementation-dependent | Included in the documented TRL implementation |
The reward row is identical on purpose. Algorithm names do not determine where scores come from; the training setup does.
How one GRPO update works
The following sequence describes the documented GRPO Trainer behavior for a single prompt. It is the default normalization in the TRL 0.18.0 GRPO Trainer documentation, as hosted in NVlabs’ GDPO repository (GRPO Trainer documentation).
Rank #2
- Sample a group of G completions from the current policy for the same prompt.
- Score each completion. The score comes from whichever reward source the run is configured to use: a reward model or a custom reward function.
- Compute the group mean and standard deviation of those rewards.
- Set each completion’s advantage to its reward minus the group mean, divided by the group standard deviation:
A_i = (r_i - mean(r)) / std(r). - Apply a PPO-style clipped policy-ratio update using those advantages, and add a KL term that penalizes divergence from the reference policy.
Step 3 is where the critic used to be. The group supplies the baseline; the reward source supplies the values being compared. If every completion in a group receives the same reward, the centered advantages carry no ranking signal, so the update has little to learn from that prompt.
Where the reward model still fits
The same documentation supports more than one kind of reward source, so the reward model is a configuration choice rather than a fixed part of GRPO. Consider three cases:
- Learned reward model. A separate model scores each completion. This is the setup most often associated with preference-based training, and it keeps a second trained model in the pipeline even though the critic is gone.
- Custom reward function. Code scores completions against a task criterion, such as checking a final answer or running a test. No learned scorer is required, though the reward is still computed for every completion.
- Other task-appropriate scoring. The cited material describes reward computation generically, so any scorer that returns a number per completion fits the same slot.
Choosing between these is a task-design decision. Claims that all GRPO systems use a learned reward model, or that all GRPO systems avoid one, are not supported by the implementation documentation.
What GRPO does not remove
“Critic-free” does not mean the training loop has no other models or constraints. In the documented implementation, the policy objective keeps PPO-style clipping, and a reference-policy KL term keeps the trained model from drifting too far from its starting point. The reference policy is a separate model copy, conceptually distinct from the critic. Exact loss formulations and reward scaling can vary with the formulation and trainer configuration, so treat the normalization formula above as the documented default, not a universal rule.
The trade-off: less value-model training, more sampling
Dropping the critic removes a trained model and its update cost. The price is that GRPO needs several completions per prompt to form a group baseline. The DeepSeekMath paper positions GRPO as a way to improve mathematical reasoning while optimizing PPO’s memory usage. The sampling cost follows from the group mechanism described in the TRL documentation, and it is a real budget item for anyone running experiments. The same documentation discusses distributed GPU training and GPU-memory constraints, so group size and reward cost should be planned together.
Recommended Free Tools
Reading the DeepSeekMath benchmark figures
The DeepSeekMath paper reports 51.7% on the MATH benchmark for its 7B model, and 60.9% using self-consistency over 64 samples. These are the paper’s reported results for that model and evaluation setup, dated 2024. They do not show that removing the critic alone produced those scores, since the paper credits several contributors, including its data selection pipeline. The paper also describes 120B math-related tokens used for continued pretraining. That is training data scale, not a count of GRPO rollouts.
Quick Recap
Best Value
- Use scikit-learn to track an example ML project end to end
- Explore several models, including support vector machines, decision trees, random forests, and ensemble methods
- Exploit unsupervised learning techniques such as dimensionality reduction, clustering, and anomaly detection
- Dive into neural net architectures, including convolutional nets, recurrent nets, generative adversarial networks, autoencoders, diffusion models, and transformers
- Use TensorFlow and Keras to build and train neural nets for computer vision, natural language processing, generative models, and deep reinforcement learning
Common misreadings to avoid
- “GRPO removes the reward model.” It removes the critic. Reward scoring is still required for each completion.
- “GRPO always uses a learned reward model.” The documented implementation also supports custom reward functions.
- “Critic-free means no other model.” The reference policy and the KL term remain in the documented setup.
- “The benchmark gain came from critic removal.” The paper attributes results to several components, not one change.
“
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
