AI-Native Series Β· Reinforcement Learning
The Scale Furnace
1-minute takeaway β what you'll walk away with
At scale, RL algorithms stop failing loudly and start failing silently. Policy entropy falls monotonically, sampled outputs converge on each other, exploration ends β and the loss curve stays smooth the whole way. Then groups start returning identical rewards, std(r) hits zero, the advantage hits zero, and your gradient is gone. Meanwhile every denominator in your loss is quietly deciding which examples matter: Dr.GRPO found that GRPO's length normalisation specifically inflates the length of incorrect answers, and that dividing by std(r) over-weights low-variance questions [2]. DAPO raised the upper clipping bound and resampled dead groups, reaching 50 on AIME 2024 against 47, at half the training steps [1]. If you log one extra scalar in an RL run, log entropy. Loss will not tell you.
Everything in the first three episodes assumed the algorithm was the hard part. At scale it is not. At scale, the hard part is that your run can stop learning without anything looking wrong.
The one idea: silent failure has three shapes
Entropy collapse. Policy entropy β how spread out the model's probabilities are β decreases monotonically through training. Sampled outputs converge toward near-identical solutions. Exploration terminates before the policy has found the strategies it was capable of finding. The run does not crash. It quietly stops discovering anything, which is worse, because the loss curve still looks fine.
Zero-variance groups. GRPO computes the advantage from a group of sibling rollouts. If every sibling gets the same reward β all correct on an easy prompt, all wrong on a hard one β the standard deviation is zero, the advantage is zero, and that entire group contributed nothing to the gradient. You paid full compute for it.
Denominators. Every normalisation term in your loss is a decision about which examples matter, and it was almost certainly chosen for numerical convenience by someone who did not intend to make that decision.
The numbers
DAPO pairs two interventions [1]. Clip-Higher decouples the upper and lower clipping bounds, raising the upper one so that low-probability exploratory tokens are not suppressed β under symmetric clipping they are hit hardest, because their ratios move most. Dynamic Sampling resamples prompts where every rollout in the group earned the same reward, which is to say: it refuses to spend compute on groups that cannot produce a gradient.
The result: 50 points on AIME 2024 with Qwen2.5-32B, against 47 for DeepSeek-R1-Zero-Qwen-32B, using 50% of the training steps [1].
Read that carefully, because the interesting number is not the 50. Three points of accuracy is an argument. Halving the compute to get there is the finding. The intervention did not make the model smarter. It stopped the optimizer from throwing away its own gradient signal.
The mechanism one layer down: your denominator is a value judgement
Dr.GRPO [2] found two distinct biases in standard GRPO, and they are worth stating precisely because they pull in different directions.
Length bias. The length normalisation inflates the length of incorrect responses specifically. The model learns to write longer when it is wrong β the exact opposite of anything anyone intended, and on a dashboard it looks like "the model is reasoning more".
Response-level difficulty bias. Dividing the advantage by std(r) over-weights questions whose rollouts happened to have low variance. Dr.GRPO removes the scaling so every question is treated equally.
Removing both terms reached state of the art on Math-7B in 27 hours on 8ΓA100 [2] β a number worth holding onto, because it says this class of fix is a correction, not a scale-up.
The lesson generalises well past this paper. Every denominator in your loss is a policy decision about which examples matter, and nobody writes that decision down.
Predict before you read on
Standard GRPO divides each advantage by the group's reward standard deviation, and normalises the loss by response length. Both look like ordinary numerical hygiene. One of them produces a specific, visible pathology in the training logs. Which, and what does it look like?
(a) Length normalisation makes correct answers shorter. (b) Length normalisation makes incorrect answers longer. (c) std normalisation makes hard questions dominate. (d) std normalisation makes low-variance questions dominate.
Two of these are true, and they are not the two most people pick.
Failure Room: turn the exploration knob the wrong way
Take a working GRPO run and make one change β tighten the upper clipping bound instead of raising it.
What breaks, in order: low-probability tokens get suppressed hardest; entropy falls faster than baseline; completions within a group start looking identical; std(r) goes to zero; the advantage goes to zero; and the loss curve looks perfectly healthy the entire time.
That last step is the point. No error, no spike, no NaN. A smooth, professional-looking curve for a model that stopped learning thousands of steps ago.
If you log one extra scalar in an RL run, log policy entropy. If you log two, log the fraction of groups with zero reward variance β that is your gradient, disappearing.
Reality Mission
Pick any training or evaluation loop you run this week β an RL run, a fine-tune, even a nightly benchmark. Find every denominator and write down what it says about which examples matter.
Mean over tokens? You decided long examples matter more. Mean over sequences? You decided they matter the same. Dividing by a standard deviation anywhere? You decided low-variance cases matter more, and you almost certainly did not mean to. You will find at least one you cannot justify.
What this costs you, honestly
The constants do not transfer. DAPO's clipping bounds were tuned for that model, that task family, that sequence length. The mechanism is reusable; the numbers are not.
These fixes interact. Clip-Higher, dynamic sampling and a changed denominator all touch exploration. Stacking them is not additive, and papers report them mostly in isolation.
And a single run is not a measurement. Every number in this episode comes from a specific setup; whether it reproduces on yours is an experiment you have to run, with enough seeds to detect the effect you are claiming.
Where this goes
Episode 5 is what happens when the completion stops being text and becomes a trajectory through a world that answers back β where the harness stops being scaffolding around the policy and becomes part of it.
Episode 4 of Intelligence Engineering Adventures, Season 1 β The Consequence Engine. Claims in the series source are tagged by class β definition, derivation, evidence, engineering choice, open question β and a metaphor may introduce a claim but never serves as evidence for it. Each episode ships a CPU-runnable lab. This article contains no material from any employer or client. β Paul Jialiang Wu Β· agentic-portfolio-lovat.vercel.app
References
- Yu, Q. et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. Clip-Higher and Dynamic Sampling; 50 points on AIME 2024 with Qwen2.5-32B against 47 for DeepSeek-R1-Zero-Qwen-32B, at 50% of the training steps. arxiv.org/abs/2503.14476
- Liu, Z. et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective (Dr.GRPO). The length bias and the std-normalisation difficulty bias; state of the art on Math-7B in 27 hours on 8ΓA100. arxiv.org/abs/2503.20783
- Zheng, C. et al. (2025). Group Sequence Policy Optimization (GSPO). Sequence-level importance ratios for MoE models and long-form reasoning. arxiv.org/abs/2507.18071
- Chen, A. et al. (2025). MiniMax-M1 (CISPO). Clipping the importance-sampling weight rather than the update. arxiv.org/abs/2506.13585
- Shao, Z. et al. (2024). DeepSeekMath (GRPO). The algorithm all of the above are patching. arxiv.org/abs/2402.03300
- Schulman, J. et al. (2017). Proximal Policy Optimization Algorithms. The clipping this episode decouples. arxiv.org/abs/1707.06347