AI-Native Series · Reinforcement Learning
The Probability Forge
1-minute takeaway — what you'll walk away with
A policy gradient does not teach a model a fact — it moves probability mass. Push up what worked, and everything else drops a little. That single sentence makes the whole algorithm family legible, because it exposes the one thing they all argue about: how surprised should we have been? Reward a lucky trajectory and you have made luck likelier. Subtract an expectation first and you reward only the part that beat the forecast. REINFORCE subtracts nothing. RLOO subtracts the siblings' average. Actor-critic learns the expectation with a second model. GRPO fires that model and uses the group. Removing the baseline does not make the estimator wrong — it stays unbiased. It makes it expensive, and a variance problem does not raise an error, which is exactly why this took the field years rather than an afternoon.
Here is a thing that sounds like a technicality and is actually the whole subject.
A policy gradient update does not install a fact in a model. It redistributes probability mass. A rollout goes well, you raise the probability of the tokens that produced it, and — because the distribution must still sum to one — everything else drops a little.
That framing immediately produces the field's oldest problem. Suppose a model solves something correctly one time in ten, and you train on the success. You have not necessarily taught it the method. You may have simply made that particular lucky path more likely.
The one idea: subtract what you should have expected
Everything in this episode is one repair, applied five ways.
Was this outcome better than what we should have expected?
Reward the raw return and you reward luck. Reward the difference between the return and its expectation and you reward only the part that beat the forecast. That difference has a name — the advantage — and every algorithm below is a different answer to the question of where the expectation comes from.
REINFORCE [1] subtracts nothing. It is unbiased and it is honest and it shakes. RLOO [2] samples several completions for the same prompt and uses the others' average as the expectation — the siblings are the baseline. Actor–critic hires a second network to predict the expectation. GRPO [3] fires that second network and goes back to the group, which is cheaper in memory and, as Episode 4 shows, buys a fresh family of failures.
Five names. One term.
The mechanism one layer down: bias is not the problem, variance is
This is the part I got wrong for a long time, and it is worth being precise about.
Removing the baseline does not make the gradient estimator incorrect. Subtracting any quantity that does not depend on the action leaves the estimator unbiased — the expected update is the same. What changes is the variance of that update.
And a variance problem does not announce itself. There is no error, no NaN, no failed assertion. You get a learning curve that is jagged instead of smooth, two random seeds that disagree, and a run that needs four times the samples to reach the same place. Every one of those reads as bad luck or a bad hyperparameter, which is precisely why the fix took the field years instead of an afternoon.
Predict before you read on
Run policy gradient twice on an easy task — once with a mean baseline, once without — same seeds, same everything else. What happens to the no-baseline arm?
(a) It fails to learn. (b) It learns to a worse final score. (c) It learns to the same place, slower and more erratically.
The answer is (c), and it is the uncomfortable one, because an unbiased-but-noisy method looks like a working method on anything small enough to brute-force. The lab in this episode makes that visible: watch the variance band, not the final number.
The speed limiter: why PPO exists
Once you can estimate the advantage, a second problem appears. You have sampled behaviour from the old policy, and you want to make several updates from it. But after the first update, the policy that generated the data is no longer the policy you are improving.
The importance-sampling ratio measures how far apart they have drifted. PPO's contribution is a speed limiter: clip that ratio, so a single batch can never push the policy further than a trust region allows. That is it. The clipped surrogate objective is a governor bolted onto a machine that would otherwise over-rev on its own stale data.
Notice what has happened to the acronyms. REINFORCE, RLOO and GRPO are arguments about the expectation. PPO is an argument about how far one update may travel. They are not competitors; they are different bolts.
What this costs you, honestly
The "one subtraction" frame is a compression and it drops real things.
GAE is not in it. The generalised advantage estimator is a bias–variance dial across time steps, and reducing it to "the expectation" hides the tradeoff it exists to expose.
Two different KLs get conflated. The old policy and the reference policy are distinct objects doing distinct jobs — one bounds the update, one bounds the drift from the model you started with. Calling both "KL" is how people end up regularising toward the wrong thing.
And unbiased is not the same as safe. An estimator can be unbiased and still concentrate probability on a lucky region faster than your evaluation can catch it.
Run it yourself, in about fifteen minutes
The lab runs three arms on the same task and seeds: no baseline, a mean baseline, and — if you extend it — a learned one. It plots the variance of the updates, not just the return.
One warning about your own extension, which applies far beyond this lab: if you compare arms across three seeds and see no difference, you have learned about your budget, not about baselines. Variance claims need enough seeds to detect the effect you are claiming. Write down how many you would need before you run it.
Where this goes
Every algorithm here optimises a number someone chose. None of them can tell you whether that number was the right one to chase.
Episode 3 is about what happens when the scoreboard itself is wrong — and about the fact that an economist described that failure mode in 1975, half a century before anyone trained a reward model.
Episode 2 of Intelligence Engineering Adventures, Season 1 — The Consequence Engine. Claims in the series source are tagged by class — definition, derivation, evidence, engineering choice, open question — and a metaphor may introduce a claim but never serves as evidence for it. Each episode ships a CPU-runnable lab. This article contains no material from any employer or client. — Paul Jialiang Wu · agentic-portfolio-lovat.vercel.app
References
- Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning. The original REINFORCE. link.springer.com/article/10.1007/BF00992696
- Ahmadian, A. et al. (2024). Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. The leave-one-out baseline for LLM feedback learning. arxiv.org/abs/2402.14740
- Shao, Z. et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. Introduces GRPO. arxiv.org/abs/2402.03300
- Schulman, J. et al. (2017). Proximal Policy Optimization Algorithms. The clipped surrogate objective. arxiv.org/abs/1707.06347
- Schulman, J. et al. (2015). High-Dimensional Continuous Control Using Generalized Advantage Estimation. GAE, and the bias–variance dial. arxiv.org/abs/1506.02438
- Sutton, R. & Barto, A. (2018). Reinforcement Learning: An Introduction, 2nd ed. incompleteideas.net/book/the-book-2nd.html