AI-Native Series · Reinforcement Learning

The Consequence Engine

By Paul Jialiang Wu · agentic-portfolio-lovat.vercel.app · 2026-08-24 · Episode 1 of 5

Cover: white background with a black rail down the left edge. The label 'AI-NATIVE SERIES · REINFORCEMENT LEARNING' sits above a serif headline, 'The Consequence Engine', and the line 'Every RL acronym you have ever been afraid of changes one or two boxes of the same seven-box loop.' Below, seven connected boxes read TASK, POLICY, ROLLOUT, REWARD, CREDIT (filled black), UPDATE, EVAL, with a dashed line closing the loop back to TASK. Three grey cards at the bottom read: GRPO CHANGES Credit — siblings replace the critic; DAPO CHANGES Rollout + Update — 50 AIME vs 47, half the steps; R1-ZERO CHANGED Reward only — 15.6 percent to 71.0 percent on AIME 2024. — The loop is the whole subject. The acronyms are edits to it.

1-minute takeaway — what you'll walk away with

Reinforcement learning for language models looks like a zoo: REINFORCE, RLOO, PPO, GRPO, DAPO, Dr.GRPO, GSPO, CISPO, RLVR. It reads like nine subjects. It is one loop with seven boxes — task, policy, rollout, reward, credit, update, evaluation — and almost every acronym modifies one or two boxes, never the whole thing. Line them up and five of seven are arguing about a single term: how much credit a given behaviour deserves. That is not a simplification for beginners; it is the map that tells you which paper to read when something breaks. The proof it matters: DeepSeek-R1-Zero moved AIME 2024 from 15.6% to 71.0% by changing the reward box and the credit box, with no supervised fine-tuning and no learned critic. Same seven boxes. Two edits.

There is a particular kind of tiredness that comes from reading about reinforcement learning for language models.

You start with PPO. Then someone says PPO is out, it's GRPO now. Then GRPO has a bias, so it's Dr.GRPO. Then GRPO collapses at scale, so it's DAPO. Then GSPO, because the ratio was at the wrong granularity. Then CISPO, because the clipping deleted the interesting tokens. Then TIS, because the training engine and the inference engine disagree about what probability they assigned.

Nine acronyms in, the honest internal monologue is: am I supposed to have read nine papers to understand one idea?

No. You are supposed to have one picture.

The one idea: seven boxes, and everyone edits two of them

Pretraining and supervised fine-tuning ask a model the same question in two different accents: can you predict or imitate the desired output?

Reinforcement learning asks something categorically different:

Can you act, observe the consequences, and change the probability of your future behaviour so that useful outcomes become more likely?

That is the entire subject in one sentence, and it runs as a loop with seven jobs. Task — what situation is the model in. Policy — what probabilities does it assign to possible actions. Rollout — what did it actually do when we sampled it. Reward — how good was the consequence. Credit — which of its behaviours deserves the praise or the blame. Update — change the parameters. Evaluation — did that change generalise, or did it just please the scoreboard.

Now the useful part. Take the nine acronyms and ask, for each one, which box it touches.

Infographic titled 'Which box does each acronym actually change?'. Across the top: the seven boxes TASK, POLICY, ROLLOUT, REWARD, CREDIT, UPDATE, EVAL. Seven rows below, each an algorithm with a one-line gloss and a filled dot under the box or boxes it modifies. REINFORCE, the honest baseline — dot under CREDIT. RLOO, siblings as baseline — CREDIT. PPO, a speed limiter — CREDIT and UPDATE. GRPO, fire the critic — CREDIT. DAPO, keep exploring — ROLLOUT and UPDATE. Dr.GRPO, fix the denominator — CREDIT. RLVR slash R1-Zero, let reality grade it — REWARD. A footer card reads: Seven algorithms. Nobody touches TASK, POLICY or EVAL. Five of the seven argue about CREDIT — which is to say: about one term in one equation. — Nobody touches task, policy or evaluation. Five of seven are arguing about credit.

Five of seven land on the same box. They are not nine subjects. They are one long argument about a single question — was this outcome better than what we should have expected? — conducted by people who kept finding new ways for the previous answer to break.

That observation will save you from what I can only describe as acronym-induced spiritual dehydration.

Why the loop is not a teaching device

It would be reasonable to suspect the seven boxes are a friendly diagram invented for articles like this one, and that real systems are messier. Here is the counter-example that changed my mind.

DeepSeek-R1-Zero took a base model and improved its AIME 2024 pass@1 score from 15.6% to 71.0% [1]. Roughly four-fold, on a competition maths benchmark. The final R1 model reaches about 79.8% after a further supervised and RL pipeline [1].

What did they change? Not the architecture. Not the pretraining. They changed the reward box — replacing a learned neural critic with rule-based signals: is the final answer, in its required box, correct, and does the output follow the required <think>…</think><answer>…</answer> format [1]. And they changed the credit box, using GRPO, which drops the learned value model and computes the baseline from a group of sibling rollouts instead [2].

Two boxes. Four-fold improvement. The loop is load-bearing.

The mechanism, one layer down: RL moves probability mass

Here is the thing that took me embarrassingly long to internalise, and which makes every later acronym legible.

A policy gradient update does not teach the model a fact. It redistributes probability mass. When a rollout goes well, the update raises the probability of the token sequence that produced it, and — because probabilities must sum to one — lowers everything else a little. That is all it does.

Which immediately tells you the danger. Suppose the model solves a problem correctly once in ten attempts, and you train on that one success. Did it learn the reasoning procedure? Or did it get lucky, and you have now made the lucky path more likely?

The outcome alone cannot tell you. Same reward, two completely different learning events, indistinguishable from the reward signal. This is the failure that the entire credit column of that infographic exists to address — and the first repair is a question so simple it sounds like nothing:

Was this result better than what we should have expected?

Subtract an expectation, and you stop rewarding luck. That subtraction is the baseline. Learn the expectation with a second model and you have a critic. Compute it from the group instead and you have GRPO. Argue about how to normalise it and you have Dr.GRPO. It is one term, all the way down.

Predict before you read on

Before the next section, commit to an answer — the point of a prediction is that it can be wrong.

Your model solves a maths prompt correctly one time in ten. You fine-tune on that single successful trace. Which happened: (a) it learned the reasoning algorithm, (b) it memorised a narrow trajectory, or (c) it raised probability around a region that happened to work?

The research-grade answer is that you cannot tell from the outcome, and any claim otherwise is a claim about your evaluation, not about your model. Separating those three hypotheses needs pass@k, deliberate exploration measurement, and held-out distribution shift. If your eval reports a single accuracy number, all three look identical.

What this costs you, honestly

The seven-box map is a compression, and compressions lose things. Three that matter.

It flattens genuine mathematical differences. PPO's clipped surrogate objective and GRPO's group-relative advantage are not interchangeable just because both live in the credit column. The map tells you where to look; it does not tell you the derivation.

Two boxes are missing on purpose. Real systems have a training engine and an inference engine that can assign different probabilities to the same token, and an asynchronous scheduler deciding how stale a rollout may be before it is discarded. Those live between the boxes and cause a genuinely nasty class of bug.

And the map cannot tell you what to optimise. Every box downstream of reward is machinery for pursuing a number you chose. Choosing it badly is not a bug the credit column can fix — which is the subject of Episode 3, and the oldest problem in the series.

Run it yourself, in about fifteen minutes

The claim "a baseline reduces variance" is the kind of thing that is easy to nod along to and hard to actually believe until you watch it. The season ships a CPU-runnable lab per episode — no GPU, no model downloads, plain PyTorch.

Episode 1's lab runs policy gradient with and without a baseline on the same task and the same seeds, and plots the variance of the updates. What you are looking for is not that the baseline version scores higher. It is that the no-baseline version eventually gets there too, while shaking violently the whole way. Removing the baseline does not make the estimator wrong. It makes it expensive. That distinction is why baselines took years to become standard, and it is invisible in any write-up that only reports the final number.

Where this goes

Five episodes, one loop, each one earning the right to the next.

Episode 2 opens the credit box properly — the log-derivative trick, baselines, advantages, and the road from REINFORCE to PPO to GRPO as one continuous argument. Episode 3 asks what happens when the scoreboard itself is wrong, and finds that economists wrote that law down in 1975. Episode 4 is what happens when elegant algorithms meet long sequences and stale rollouts. Episode 5 is when the model stops producing text and starts taking actions in a world that answers back.

One thing to carry out of this one: when the next acronym arrives — and it will, probably this month — do not ask what it is. Ask which box it edits. That question is answerable in about ninety seconds from any abstract, and it tells you whether the paper is relevant to the thing that is currently broken in your run.

Episode 1 of Intelligence Engineering Adventures, Season 1 — The Consequence Engine. Runnable labs, diagrams and the full five-episode arc are in the series source. Claims in this piece are tagged in the source by class — definition, derivation, evidence, engineering choice, open question — and a metaphor may introduce a claim but never serves as evidence for it. This article contains no material from any employer or client. — Paul Jialiang Wu · agentic-portfolio-lovat.vercel.app

References

Guo, D. et al. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Source of the AIME 2024 pass@1 figures (15.6% → 71.0% for R1-Zero; ≈79.8% for R1) and of the rule-based accuracy + format reward design. arxiv.org/abs/2501.12948
Shao, Z. et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. Introduces GRPO — the group-relative baseline that removes the learned critic. arxiv.org/abs/2402.03300
Yu, Q. et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. Clip-Higher and Dynamic Sampling; 50 points on AIME 2024 with Qwen2.5-32B against 47 for DeepSeek-R1-Zero-Qwen-32B, at 50% of the training steps. arxiv.org/abs/2503.14476
Liu, Z. et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective (Dr.GRPO). Identifies the length bias and the std-normalisation difficulty bias in GRPO. arxiv.org/abs/2503.20783
Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning. The original REINFORCE. link.springer.com/article/10.1007/BF00992696
Sutton, R. & Barto, A. (2018). Reinforcement Learning: An Introduction, 2nd ed. The formal definition used here. incompleteideas.net/book/the-book-2nd.html
Ahmadian, A. et al. (2024). Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. arxiv.org/abs/2402.14740