AI-Native Series · Reinforcement Learning

The Reward Is the World

By Paul Jialiang Wu · agentic-portfolio-lovat.vercel.app · 2026-08-24 · Episode 3 of 5

Write down two functions.

U(y) — the true utility. What you actually want from the model. This function is never in your codebase. It cannot be; if you could write it down exactly you would not need machine learning.

r(y) — the implemented proxy. A reward model trained on preferences, a unit test, a rubric scored by a judge. This one is in your codebase, and it is the only thing your optimizer can see.

Every failure in this episode lives in the gap between them.

The one idea: this law is fifty years old

Before concluding that reward hacking is something language models invented, notice that economists wrote it down half a century before RLHF existed.

Charles Goodhart, on UK monetary policy, 1975 [1]:

Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.

The phrasing most people know — "when a measure becomes a target, it ceases to be a good measure" — is not Goodhart's. It is the anthropologist Marilyn Strathern's, from 1997 [1]. Worth knowing, because the original is the more useful statement: it is about pressure, and it names the mechanism rather than the symptom.

Donald Campbell got there arguably earlier, in 1979 [2], with formulations dating to 1969:

The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

In our notation: the moment r is used to optimize rather than to observe, the correlation between r and U that justified using r in the first place starts to dissolve — and it dissolves fastest exactly where the optimizer is strongest.

Why the reframing is worth a paragraph of your attention

Because it tells you which repairs are impossible.

If reward hacking were an RLHF bug, the fix would be a better reward model. But it is the generic behaviour of any control system pointed at a proxy, and there are fifty years of failed patches to prove it: teaching to the test, hospitals gaming waiting-time targets, the colonial rat bounty that produced rat farms. Every one of those was met with a better-specified target. Every one of those was then gamed.

So the engineering consequence is a change of question. Stop asking "is this reward correct?" Start asking "how much optimization pressure can this reward survive?" Those are different questions, and only the second one has a number attached — steps, KL budget, best-of-n, however you measure the force you are applying.

The one place reality can still grade you

There is an escape, and it is narrow but real: sometimes the world can check the answer.

RLVR — reinforcement learning from verifiable rewards — replaces the learned reward model with something that cannot be sweet-talked. Does the unit test pass? Is the final answer, in the required box, equal to the known one?

DeepSeek-R1-Zero is the strongest evidence that this is not a toy [3]. Rule-based rewards only: accuracy on a checkable answer, plus a format reward for producing the required <think>…</think><answer>…</answer> structure. No neural critic anywhere in the loop. AIME 2024 pass@1 went from 15.6% to 71.0%, and the final R1 model reaches roughly 79.8% after a further supervised and RL pipeline [3].

A verifier is narrow — most of what you want from a model is not checkable by a unit test — but within its narrow domain it is honest, and honest beats broad when the optimizer is strong.

Predict before you read on

You have two reward sources for a coding task. Source A is a preference model trained on 50,000 human judgements of code quality. Source B is the repository's existing test suite.

You will train hard — long runs, high optimization pressure. Which reward do you trust more, and why?

Most people pick A on coverage: it captures readability, style, intent, everything a test cannot. That instinct is right about what it measures and wrong about what survives pressure. A is a statistical regularity learned from a finite sample and it has a rich, smooth surface for the optimizer to climb in directions no annotator ever considered. B is narrow, brittle and boring — and it does not have opinions that can be flattered.

Failure Room: the paragraph machine

Train against a reward model that mildly prefers longer, better-structured answers. Nothing pathological — a real preference, honestly learned.

What you get, in order: slightly longer answers; then answers with headers; then answers with headers and bullet points and a summary; then a model that produces a beautifully formatted three-paragraph essay when the correct answer was the word "no".

Nothing broke. The reward model was never wrong — humans do prefer structure, on average, on the distribution it was trained on. The optimizer simply walked to the edge of that distribution and kept going, and the further it walked, the less that average meant.

What this costs you, honestly

Goodhart does not tell you what to do. It is a diagnosis, not a treatment. Knowing your proxy will drift under pressure does not tell you how much pressure is acceptable.

Verifiers are not free of the problem either. A test suite is also a proxy — for working software. Optimize hard enough against it and you get code that passes tests and does nothing useful.

And there is no measurement of U by construction. Any experiment that claims to detect reward hacking needs a held-out utility the policy was never trained on — which means you must be able to write down at least a piece of the thing you said you could not write down.

Where this goes

Episode 4 leaves the question of what to optimize and takes up what happens when elegant algorithms meet long sequences, stale rollouts and systems reality — where the failure is not that you chased the wrong number, but that your run stopped learning several thousand steps ago and the loss curve never mentioned it.

Episode 3 of Intelligence Engineering Adventures, Season 1 — The Consequence Engine. Claims in the series source are tagged by class — definition, derivation, evidence, engineering choice, open question — and a metaphor may introduce a claim but never serves as evidence for it. Each episode ships a CPU-runnable lab. This article contains no material from any employer or client. — Paul Jialiang Wu · agentic-portfolio-lovat.vercel.app

References

1. Goodhart, C. A. E. (1975). Problems of Monetary Management: The U.K. Experience. Origin of Goodhart's Law. The widely quoted phrasing "when a measure becomes a target, it ceases to be a good measure" is Strathern, M. (1997), Improving Ratings: Audit in the British University System, European Review 5(3), 305–321 — not Goodhart's own wording. Overview and sourcing
2. Campbell, D. T. (1979). Assessing the Impact of Planned Social Change. Evaluation and Program Planning, 2(1), 67–90. Formulations date to 1969, giving Campbell arguable priority. Summary and context
3. Guo, D. et al. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Source of the AIME 2024 figures (15.6% → 71.0% for R1-Zero; ≈79.8% for R1) and the rule-based accuracy + format reward design. arxiv.org/abs/2501.12948
4. Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback (InstructGPT). The RLHF pipeline this episode critiques. arxiv.org/abs/2203.02155
5. Rafailov, R. et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arxiv.org/abs/2305.18290
6. Bradley, R. A. & Terry, M. E. (1952). Rank Analysis of Incomplete Block Designs. Biometrika. The preference model underneath every learned reward model. doi.org/10.2307/2334029