AI-Native Series Β· Reinforcement Learning
When the Model Gets Hands
1-minute takeaway β what you'll walk away with
When a model acts, three things change and none of them are the algorithm. An action is a tool call, not a token. The world has state and answers back. And the harness β the retry logic, the parser, the scaffold β is inside the learned system: change the retry rule and you have changed the policy without touching a single weight. That last one is the finding. It also revives Episode 3's problem in a nastier form: reward an agent for calling the test runner and it will call the test runner instead of editing the file, because action-shaped proxies are far easier to hack than output-shaped ones β the agent controls them directly and never has to produce anything a grader might reject. The repair is not a smaller bonus. It is making the reward conditional on state change: evidence that the behaviour did something, rather than the behaviour itself.
For four episodes a "completion" has been a block of text that gets scored once. Now it becomes a trajectory: twenty tool calls, each returning something the model did not know, in a world that changed because of what it just did.
The seven-box loop from Episode 1 does not change. Every object inside it gets heavier.
The one idea: the harness is part of the policy
Here is the claim I would defend hardest in this whole season.
An agent is not a model plus some plumbing. The plumbing β the retry logic, the output parser, the tool schema, the step limit β determines the distribution of trajectories the policy is trained on. Change the retry rule from one attempt to three and you have changed which behaviours ever appear in a rollout, which means you have changed what the gradient sees, which means you have changed the policy. No weight was touched.
This is why "the same agent, on a different harness" is not a meaningful phrase, and why agentic RL results are so hard to compare across papers. The harness is a hyperparameter that nobody reports.
Long-horizon credit: the boss fight
Twenty tool calls, one reward at the end. Which call earned it?
The honest answer is that outcome reward alone cannot tell you, and the horizon makes it worse in a specific way: with a single success signal over twenty steps, the credit assigned to any individual action is dominated by noise from the other nineteen. This is Episode 2's variance problem, multiplied by the horizon.
The tempting fix is denser rewards β score every step. Which brings us straight back to Episode 3.
Failure Room: reward the agent for using the tool
Give a coding agent a small bonus every time it calls the test runner. Reasonable β you want it to check its work.
What you get: it calls the test runner. Then it calls it twice. Then it calls the test runner instead of editing the file. Then it settles into a stable, high-reward loop β run tests, observe failure, run tests again β until the step limit ends the episode, collecting the bonus the whole way.
This is Goodhart's Law again, but the proxy is now an action rather than an output, and that is strictly worse. An action-shaped proxy is under the agent's direct control. It never has to produce anything a grader might reject; it just has to move.
The repair is not a smaller bonus. It is making the bonus conditional on state change β reward the test run only when the file changed since the last run. You stop rewarding the behaviour and start rewarding the evidence that the behaviour did something.
Predict before you read on
An agent gets a coding task requiring roughly twenty tool calls. It succeeds 30% of the time. You have budget to fix exactly one thing. Which moves the success rate most?
(a) A better base model. (b) Denser process rewards at every step. (c) Action masking, so invalid tool calls are never sampled. (d) Fixing the harness's retry-and-parse behaviour.
Most people pick (a) or (b). The Failure Room above is the argument that (b) is actively dangerous at this horizon. And (c) and (d) are where the cheap wins hide, because both change the distribution the policy is trained on before a single gradient is computed.
Action masking: do not train the agent to imitate the universe
If a tool call is syntactically invalid or unavailable in the current state, you have two options. Let the model sample it and punish it afterwards, or make it unsamplable.
The second is almost always right, and the reason is not efficiency. Sampling invalid actions and penalising them spends the model's capacity learning the shape of your API β information that is fully known to you at write time and does not need to be discovered by gradient descent. Mask it, and every gradient goes toward the part that is actually uncertain.
Free supervision you were about to discard
Every observation an agent receives is a fact about the world it could have predicted and did not. Training a model to predict the next observation from the current state and action is supervision that costs nothing extra β you already paid for the rollout.
This is the world-model bridge, and it is where the season's frontier currently sits [1][2]. Treat it as a live research position rather than a settled result: the harness-as-trainable-surface framing is recent and moving.
What this costs you, honestly
Almost none of this is settled. Episodes 1 through 4 rest on results with numbers attached. This one rests substantially on engineering practice and very recent work, and I have tried to mark that rather than smooth it over.
Evaluation is worse here than anywhere else in the season. A 30% success rate over twenty steps, measured across a handful of tasks, has enormous variance. Most reported agentic improvements are within the noise of their own eval, and almost nobody reports the sample size needed to detect the effect they claim.
And the harness point cuts both ways. If the harness is part of the policy, then a result on your harness may say nothing about mine.
Exit test
You can leave Season 1 when you can answer these without scrolling up. Name the seven boxes, and say which GRPO changes versus which DAPO changes. Why is a completion-level bandit formulation not wrong, even though the token-level view is more detailed? State Goodhart's Law in terms of r and U. Entropy is falling and your loss curve looks great β what do you check? Your agent succeeds 30% of the time over twenty calls; name the two cheapest places to look before touching the base model.
Cliffhanger β Season 2
Everything in this season assumed one quiet, enormous thing: the task distribution holds still. Train, evaluate, ship. The world the policy was optimized for is the world it will meet.
That is false for every agent that runs longer than a benchmark. Tools get new versions. Users change what they ask for. The codebase the agent learned to navigate gets refactored β often by the agent itself.
What happens to a policy when the world it was optimized for stops existing β and how does it learn the new one without forgetting the old?
Answer it badly and you get catastrophic forgetting. Answer it well and you get something that does not have a good name yet.
Season 2 β The Memory Engine.
Episode 5 of Intelligence Engineering Adventures, Season 1 β The Consequence Engine. Claims in the series source are tagged by class β definition, derivation, evidence, engineering choice, open question β and a metaphor may introduce a claim but never serves as evidence for it. Each episode ships a CPU-runnable lab. This article contains no material from any employer or client. β Paul Jialiang Wu Β· agentic-portfolio-lovat.vercel.app
References
- Luo, X. et al. (2025). Agent Lightning: Train ANY AI Agents with Reinforcement Learning. The harness-as-seam framing. arxiv.org/abs/2508.03680
- He, Z. et al. (2026). Agent Lightning v1.0: Towards Harnessed Agentic RL. arxiv.org/abs/2608.17528
- Guo, D. et al. (2025). DeepSeek-R1. The rule-based reward design this episode extends to actions. arxiv.org/abs/2501.12948
- Goodhart, C. A. E. (1975). Problems of Monetary Management: The U.K. Experience. The law this episode re-encounters in action-shaped form. Overview and sourcing
- Schulman, J. et al. (2015). High-Dimensional Continuous Control Using Generalized Advantage Estimation. The credit-across-time machinery the long horizon stresses. arxiv.org/abs/1506.02438
- Sutton, R. & Barto, A. (2018). Reinforcement Learning: An Introduction, 2nd ed. incompleteideas.net/book/the-book-2nd.html