AI-Native Series · Agent Engineering, Compared: Prompt · Context · Harness · Loop · Graph

Evaluation Engineering Is the Truth Function, Not a Leaderboard

Episode 6 of 6, and the close of the season. Five episodes each ended with a number — determinism 1.000, leakage zero, no out-of-envelope calls — and every one came out of a grader nobody audited. This is the rung that grades the grader, and why the ladder's sixth row stayed blank until now.

By Paul Jialiang Wu · agentic-portfolio-lovat.vercel.app · 2026-09-11 · Episode 6 of 6

What this episode covers, and why evaluation engineering is the sixth rung

This season has compared six named disciplines for building AI agents. Five were published and datable from the start — prompt (2020), loop (2022), graph (2023), harness (2024), context (2025). Feng et al. (arXiv:2608.21156) order them by what each lets you control — prompt, context, harness, loop, graph — which is this season's episode order, and it is not the order of their landmark years. The sixth row has been printed ??? in every episode since the first.

[DESIGN] Here is why, once, so it is on the record. When Episode 1 shipped on 2026-09-03, an external reviewer proposed that the open sixth layer should be evaluation engineering — the truth function every other layer borrows to know whether it improved — with governance, economics and organisation behind it. Episode 1 recorded that as a candidate and left the decision to me, because naming a rung you have not argued is the listicle behaviour this season exists to avoid. The argument now exists, so the call is made here: rung six is evaluation engineering. It stayed unnamed because a discipline earns its row by breaking the row below it, and you cannot show that until the rows below exist.

Every episode asked its rung four questions: where did it come from, what do you control, who drives it, which failure pushes you up. This one has a fifth, and it is why this is the finale rather than an appendix: what told you any of the previous five worked?

Who this is for. An AI engineer whose CI says green and whose users say otherwise. A founder about to put a benchmark number in a deck. An executive signing off on an agent that will touch money. A high-school builder who wants to know what "the eval" is. Prerequisite: having believed a number.

Where evaluation engineering came from, and who is driving it

[DEF] An eval is, in Anthropic's own definition, "a test for an AI system: give an AI an input, then apply grading logic to its output to measure success" — the right definition because it puts the grading logic inside it rather than beside it. Evaluation engineering is the discipline of building that grading logic so its number can be trusted, and of measuring the grading logic itself, which is the part almost everyone skips.

[DESIGN] Note what is not on that list: a leaderboard. The graders climbed the same ladder the agents did — exact match, similarity score, execution oracle, model — each step buying coverage and selling certainty. Leaderboards are the output; the discipline is the machinery behind them, and only one of the two has a screenshot.

Who drives it now. Academic groups own the critique — Princeton (Kapoor, Narayanan, Jimenez), Stanford CRFM, LMSYS/LMArena. The labs own the practice: Anthropic treats evals as CI, OpenAI ships a framework you register tests in. And the newest layer is a fast-growing literature whose entire subject is the grader — AgentJudgeBench, the voice-agent and mobile-agent judge studies. It did not exist two years ago, which is the clearest signal this rung is real.

The mental model: the recall log

One picture has carried all six episodes; here it is for the last time. Every cook climbs the same ladder — yourself, family, a restaurant kitchen, a fast-food chain, the national supply chain feeding several chains — and every AI engineer climbs a matching one: school learner, junior developer, senior developer, AI tech lead, CTO, frontier leader. Rung one was the recipe card. Rung six is the recall log.

A national food system does not learn from the plates that were fine. It learns from what came back wrong — the batch that made people ill, the supplier whose oil was not what the label said. The recall log is the only document in the system that can declare a number the rest of the system produced false.

And it has a property nobody enjoys: an empty recall log means one of two things, and only one is good news. Either the food is safe, or nobody is testing it. From inside the log those look identical — which is why a green dashboard is not evidence.

Two axes, not one. Every kitchen runs all six layers at once: a home cook has a card, a pantry, a stove, a taste test, a supplier — and a memory of the dish that made someone sick. What changes as you climb is how many layers you may touch and how rigorously you run each one: by feel, then measured, then gated, then self-correcting. The six disciplines are layers (what you control); seniority is maturity (how you run them). Evaluation is not more advanced than graph: a three-node graph with a calibrated golden set is more mature than a forty-agent topology whose only eval is a demo that went well. That lands hardest here, because evaluation is the layer most often run at maturity zero by teams sophisticated everywhere else.

The analogy may introduce that claim; it may never be the evidence for it. The pinning is in the 1:1 map below.

The vocabulary, pinned to its source

[DEF] Four words do the work here, and three are routinely used to mean their opposite.

• Golden set — inputs with outcomes a human wrote down; not outputs your model produced and you skimmed. The distinguishing property: a golden label can disagree with your system and win.
• Judge — any grading logic: a string comparison, an assertion, a test suite, or a model asked "was that right?". Zheng et al. showed the last is viable and carries measurable biases; both halves are load-bearing.
• Cohen's kappa (κ) — agreement between two labellers above chance. Raw agreement is not the metric: in this episode's first cell a coin flip agreed with a fixed list 47.7% of the time, at κ = −0.046. The conventional κ bands come from Landis and Koch (1977), offered by their authors as arbitrary benchmarks rather than derived ones; the paper is paywalled, so it is described and not quoted. I use κ ≥ 0.60 as an engineering floor because that is where their "substantial" band begins, not because 0.60 is a fact about the world.
• pass@k vs pass^k — solved at least once in k trials, versus solved in all k. Yao et al. introduced the second in τ-bench because even state-of-the-art function-calling agents were "quite inconsistent (pass^8 <25% in retail)".

[OPEN] One phrase I deliberately do not define: "the eval passed". It has no referent until you say which golden set, graded by what, reported how, at what k, and with what agreement against a human. Answer those five and you do not need the phrase; fail to, and the phrase is what hides that.

The claim, classed

[DEF] Evaluation engineering is the truth function. Every other layer's claim about itself is this layer's output, so your system's accuracy is bounded above by your grader's — the way a ruler's error bounds a measurement's.

[EVIDENCE] Three published measurements make that literal, from three directions.

One — the grader has a thumb on the scale, and it has been weighed. Zheng et al. (arXiv:2306.05685, v4) examined "the usage and limitations of LLM-as-a-judge, including position, verbosity, and self-enhancement biases, as well as limited reasoning ability", then measured the good news too: strong judges "can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans." The pairing is the finding: the judge is about as good as a person and can be moved by which answer you show first. A team that internalised only the first half is running an unmeasured instrument with a reputation.

Two — the reporting rule is a free variable worth more than most model upgrades. Yao et al. (arXiv:2406.12045) found that state-of-the-art function-calling agents — their abstract names gpt-4o — "succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail)", and proposed pass^k so consistency could not be averaged away. Nothing about the agent changes between pass@8 and pass^8. What changes is which question you asked of the same trials.

Three — the benchmark drifts from the thing you care about. Kapoor et al. (arXiv:2407.01502) is the most uncomfortable of the three because none of its findings are about models. There is "a narrow focus on accuracy without attention to other metrics", and as a result "SOTA agents are needlessly complex and costly, and the community has reached mistaken conclusions about the sources of accuracy gains." And most damningly: "many agent benchmarks have inadequate holdout sets, and sometimes none at all", producing agents that are "fragile because they take shortcuts and overfit to the benchmark in various ways."

[DERIVATION] Put the three together and you get the rule this episode exists for. A reported score is not one measurement; it is the composition of three — the agent's behaviour, the grader's agreement with a human, and the reporting rule over repeated trials. You are shown the first and told it is the whole thing. A team that improves its agent by 5% and its grader by nothing has moved its score by an unknown amount in an unknown direction.

Last month's contamination taxonomy says the same from the leaderboard's side: a benchmark score is a joint property of the model, the harness, the elicitation budget, the sampled population and contamination status — and "Leaderboards publish the model and the score, so capability and leakage stay observationally equivalent" (arXiv:2608.29463). Two of five inputs are published. That is why this episode is titled what it is.

1:1 technical map

Every element of the recall log maps to one thing you can point at in code.

[DEF] One row deserves its own paragraph — the invariant both projects are built on, and the rule my own research work already runs under: no evidence means not measured, and it never means a pass. An item the grader cannot decide is not a borderline success; it is an absence of information, and its honest record is a third state. Systems without that state do not become more accurate. They become quietly generous, in the direction of whoever is reporting.

Prediction Gate

Before you scroll, commit. Your judge agrees with your own hand labels on 79 of 100 items, on the full 100-item set, before the 20 calibration items are held out.

Q1. What is its Cohen's kappa, roughly? And is 79% agreement good?

Q2. You change nothing about the judge, but show it every item twice — candidate first, then candidate second — and average. What happens to kappa, and how many of the 100 items change verdict between the two orders?

Q3. An agent solves 100 tasks with per-task success probabilities between 0.25 and 0.95, run eight times each. Rank by size, largest first: pass@1, pass@8, pass^8.

Q4. Your judge is biased in a fixed direction. Does that hurt your measured pass@8 or your pass^8 more?

Answers in the Failure Room. If Q4 felt obvious, the notebook's cell 8 is worth four minutes.

Failure Room

Break it deliberately. A mechanism you cannot break is a claim.

Failure 1 — the aggregate that is right over a population of wrong verdicts. The lab's naive judge reports a pass rate of 0.720 against a golden 0.510, agreeing with the human on 79 of 100. Both look survivable; the kappa is 0.576 (on all 100; 0.585 on the 80 held-out), so much of that agreement was chance. An aggregate can sit near the truth while the decisions underneath it are badly wrong, and it is the decisions you ship. A pass rate with no agreement number beside it is a rumour with decimals.

Failure 2 — the grader that cannot fail anything. I built one on purpose: a judge that returns PASS for every item. Pass rate 1.000, kappa exactly 0.000. It is the food inspector who has never once failed a kitchen — a perfect record, and the reason you should never eat there. Not a strawman: it is the limiting case of every grader whose threshold was tuned until the dashboard looked good. I have written that grader, and from the inside it feels like progress.

Failure 3 — Episode 1's empty plate is still on the menu. Episode 1 ended with an agent that returns {} every time and therefore scores a Reproducibility Rate of 1.000. Five episodes later that same empty plate clears this episode's audit bench too, and the bench's own rule says why: an empty answer is never decidable, so every item lands in UNMEASURED — and the flag make negative exists to refuse is exactly the one that moves UNMEASURED into the numerator. One {} has now cleared six kitchens; it is the season's most successful agent and has never produced an output. Every metric admits a degenerate input that maximises it, and the defence is never a better metric — it is a second metric the degenerate input destroys. Determinism next to task success. Pass rate next to kappa. Coverage next to both.

Failure 4 — fixing one bias can unmask another and look like a regression. This cost me a rebuild of the lab. My first version randomised which slot each item appeared in; swapping the order then made kappa go down, 0.820 to 0.780 in that discarded build, because the random position offset had been partly cancelling the verbosity offset. Two errors pointing in opposite directions look like accuracy. If a fix makes your metric worse, the first hypothesis is not that the fix is wrong. It is that your metric was being flattered by a second error.

Failure 5 — teaching to the inspection. Goodhart's original argument — monetary aggregates, 1975, reprinted 1984 — is that a statistical regularity breaks down once used as a target for control; the crisp slogan most people quote is Marilyn Strathern's 1997 formulation about university audit. Both are paywalled, so both are described rather than quoted. The agent-scale version is measured: last month's taxonomy (arXiv:2608.29463) sorts contamination by which mitigation each type defeats, and a private test set closes only the first of five. Your golden set catches the same disease the day someone reads it before writing the agent.

Answers. Q1: κ ≈ 0.58 — 79% agreement is not good, because a coin gets ~50% free. Q2: kappa rises to 0.780 on the full 100-item set (0.726 on the 80 held-out items the rest of this episode reports), and 50 of 100 items change verdict between orders: half the graded set was decided by seating position. Q3: pass@8 (0.99) > pass@1 (0.57) > pass^8 (0.03). Q4: pass^k, by a lot — it is an AND over k judgements, so a fixed-direction bias compounds instead of averaging out. A length-biased judge read pass^8 as 0.240 when the truth was 0.030.

Notebook

Lab 06 — The truth function: judge drift and pass^k.

06_eval_truth_function.ipynb · Colab free tier · no API key · standard library only · seeded at SEED=20260911 · ~2 seconds of CPU · every cell executed before publication. The gist carries both projects too: gist.github.com/wjlgatech/752a9450…

Nine cells. The design choice that makes it worth running: the judge emits a score, not a verdict, with the biases as additive offsets. That is not a teaching simplification — it makes each fix attributable, because an offset you can name is one you can cancel and check the residue of.

1. Cohen's kappa, checked on two inputs whose answers are known: identity (1.000) and an independent coin (−0.046, at 47.7% raw agreement).
2. The golden set: 100 items with a hidden quality, a human label, and two properties — length and slot — independent of quality by construction.
3. The judge: position bias ±0.30, verbosity ±0.12, noise ±0.08. Naive verdict κ 0.576, at a pass rate of 0.720 against a golden 0.510.
4. Swapped-position re-judging. Grade in both orders and average: the offset is +d one way and −d the other, so it cancels exactly. 50 of 100 items flip verdict between orders; κ rises to 0.780 across all 100 items (0.726 on the 80 held out), and verbosity is deliberately untouched — the residue proves the fix was surgical.
5. The calibration set — 20 hand-labelled items, used not to train but to estimate the length offset with the golden label held fixed: +0.259 against a true +0.240.
6. The three regimes on the same 80 held-out items, gate applied.
7. pass@k against pass^k across k = 1, 2, 4, 8 — 100 tasks, 8 trials each.
8. The two errors composing: the same trials graded by an unbiased, a biased and a calibrated judge.
9. The Goodhart cell: the always-PASS judge, and Episode 1's {} agent's final appearance.

[EVIDENCE] The two tables the rest of this episode leans on, printed by the run:

held-out items: 80   (the 20 calibration items are excluded)
judge                      kappa     agree  measured pass
naive, one pass            0.585     64/80          0.738
+ swapped position         0.726     69/80          0.475
+ calibration set          0.899     76/80          0.537
golden labels (truth)      1.000     80/80          0.537
  gate kappa >= 0.60: naive, one pass        FAIL  (0.585)
  gate kappa >= 0.60: + swapped position     PASS  (0.726)
  gate kappa >= 0.60: + calibration set      PASS  (0.899)

100 tasks, k=8 trials each, per-task success probability drawn in [0.25, 0.95]
  k    pass@k    pass^k       gap
  1     0.570     0.570     0.000
  2     0.750     0.360     0.390
  4     0.900     0.120     0.780
  8     0.990     0.030     0.960

Two moves, no change to the judge, κ from 0.585 to 0.899. And below: the same agent, the same 800 runs, reported as 0.990 or 0.030. Both true; only one is what a person using the thing experiences.

[DESIGN] Why a stub and not a live model. A simulator reproduces where the error enters and what removes it, in seconds, free, with no key. It does not reproduce magnitudes: nothing it prints is evidence for Zheng et al.'s 80% agreement or τ-bench's pass^8 under 25%, both measured on real systems and citable only to the papers. The bridge is short — replace judge_score() with one call to any OpenAI-compatible endpoint and nothing else changes, because measurement code does not know what it is pointed at.

Projects

Two builds, from nothing — nothing beyond the standard library. Both were run before this section was written, so every number is measured, not predicted.

Project 1 — real life: the-report-card

p1_report_card.py · python3 p1_report_card.py

A habit tracker writes one line a day; a judge reads it and decides whether you kept the habit. I labelled 60 days by hand. The question is not how good is my agent — it is how good is the thing that told me how good my agent was.

the-report-card - grading the grader of a habit tracker
  golden set        : 60 days labelled by hand
  calibration       : 10 days re-read, 50 days graded
  note-length offset: estimated +0.284, true +0.320

  grader                     kappa    gate   measured streak
  before calibration         0.572    FAIL             0.680
  + swapped batch order      0.797    PASS             0.400
  + 10-day calibration       0.960    PASS             0.440
  my own labels              1.000       —             0.460

[EVIDENCE] The app told me I kept the habit on 68% of days. I kept it on 46%. It was wrong about the days, and wrong in a direction I liked, which is why I had never checked. I had been congratulating myself on a rounding error with good manners.

The two distortions are Zheng et al.'s in domestic clothes. The tracker lists newest first, so the same note scores higher at the top of a batch than at the bottom: position bias. A note containing a number — "ran 5.2 km" — reads as more credible than "went for a run", whether or not either happened: verbosity bias. Ten days re-read by hand estimated the second at +0.284 against a true +0.320.

Project 2 — enterprise: the-audit-bench

p2_audit_bench.py · make report · make gate · make negative

A support-resolution agent graded against 120 tickets whose outcome a human wrote down: 24 held out for calibration, 96 graded. The harness does not trust the judge — it measures it, publishes the number, and fails the build when the judge is not good enough or an unmeasured item is counted as a pass.

  kappa, single pass, raw   : 0.455   <- what a one-shot judge gives you
  kappa, calibration, before: 0.577   <- FAILS the 0.60 gate
  kappa, calibration, after : 1.000   on the 22 of 24 calibration tickets the harness could measure
  measured / unmeasured     : 80 / 16   (coverage 0.833)
      order-dependent               14
      inside the uncertainty band    2
  pass rate, measured only  : 0.500
  pass rate, if UNMEASURED were counted as passes: 0.583   <- the number the gate refuses

gate
  PASS  kappa on the calibration set >= 0.60         1.000
  PASS  coverage >= 0.70                             0.833 (80/96 measured)
  PASS  no UNMEASURED item is counted as a pass      0 leaked
  => green

[EVIDENCE] Hold 0.500 and 0.583 side by side. The honest pass rate is 0.500 over the 80 tickets the harness could measure; the flattering one is 0.583, and all that changed is whether 16 undecidable tickets were quietly counted as wins. An 8.3-point swing, free to anyone who never implements a third state — no lying required, just a two-valued type where the world has three values.

make gate exits 0. make negative re-introduces the defect and requires exit 1:

gate  (negative test: unmeasured items deliberately counted as passes)
  FAIL  no UNMEASURED item is counted as a pass      16 leaked, first: TCK-1029, TCK-1034, TCK-1035
  => RED          # exit 1

A gate that has never been seen red is indistinguishable from no gate — a rule I earned in a different season, now installed in a bench whose whole subject is not trusting instruments.

Build it in 30 minutes — the contract

Everything the bench needs, in the order an engineer asks for it. Inputs: N items with outcomes a human wrote. Output: exactly one of PASS · FAIL · UNMEASURED. Invariant: no evidence means not measured, never a pass. Stop rule: the gate fails the build; it does not retry.

GOLDEN   = [(item, human_label), ...]        # a human wrote these, not the model
CAL, TEST = GOLDEN[:24], GOLDEN[24:]         # calibration is HELD OUT of the score

def verdict(item, offset):
    a = judge(item, candidate_first=True)    # score, not a verdict
    b = judge(item, candidate_first=False)   # the SAME item, other order
    a, b = a - bias(item, offset), b - bias(item, offset)
    if (a >= T) != (b >= T):      return UNMEASURED, "order-dependent"
    if abs((a+b)/2 - T) < BAND:   return UNMEASURED, "inside the band"
    return (PASS if (a+b)/2 >= T else FAIL), "two orders agreed"

offset = mean(score | long, label=g) - mean(score | short, label=g)   # over CAL, per g

# CI gate (make gate)
#   kappa(verdicts, human_labels) on CAL   >= 0.60   else UNCALIBRATED
#   coverage = measured / graded           >= 0.70   else vacuously green
#   UNMEASURED items counted as passes     == 0      (make negative proves it bites)
# any line below threshold -> exit 1 -> the build fails

Deliberately missing: a live model (swap judge() for any endpoint), per-item rationales, and a third judge to break ties. That last is a decision, not an oversight — a tie is information. It says this item is not decidable by this judge, and UNMEASURED is the honest record. A tie-breaker converts a known unknown into an unknown known, which is worse.

[DESIGN] Four properties make this enterprise-shaped rather than a demo:

• Calibration items are held out of the score. A grader tuned on the items it grades has no holdout set — Kapoor et al.'s exact defect. The 24 calibration tickets never enter the pass rate.
• Coverage is gated, not merely reported. Without a floor, a harness reaches any kappa you like by declaring almost everything unmeasurable — a gate that passes by measuring nothing.
• The third state is a type, not a convention. UNMEASURED comes back from the same function that returns PASS, so no caller can forget it exists. Conventions decay; types do not.

[OPEN] What neither project proves. Both golden sets are synthetic and both judges are stubs. The audit bench's kappa of 1.000 sits on 22 items — the uncertainty band doing its job, not a perfect grader, and at n = 22 it cannot tell excellent from lucky. That is exactly the unscoped 1.000 this season has warned about twice, in my own output. Honest status: runnable and gated, not deployed and measured. What would make it real: tickets from a live queue with their actual resolutions, and a second human labeller, so inter-human kappa supplies the ceiling instead of an implied 1.000.

Same story, five exits

One spine, a door per reader. Take one decision, one action.

Reality Mission

Twenty minutes, on something you run.

1. Pick one AI output you rely on: a summariser, a router, a code reviewer, a habit tracker.
2. Collect 20 real outputs and label them yourself, before you see any score the system gave itself. Pass or fail.
3. Get the system's own verdicts on the same 20 and compute Cohen's kappa — six lines of arithmetic, in the notebook's cell 1.
4. Count how many of the 20 you could not label confidently. That count is your coverage problem.
5. If your grader is a model, show it five items in both orders and count the flips. In my lab it was 50 of 100.

If step 2 is the hard one, you have learned the most useful thing available: your team has no definition of correct that survives contact with a person, and no work on the other five rungs will supply one.

Research Challenge

Hand this to an agent and pre-register the prediction first.

Take one task family with a checkable output and build a golden set of 100 items labelled independently by two people. Report: inter-human kappa (the ceiling); a model judge's kappa against each human; the change in that kappa under (a) swapped-position re-judging, (b) a 20-item calibration correction, (c) both. Then report pass@k and pass^k for k = 1, 2, 4, 8 under the human labels and under the model judge separately, plus the cost per graded item.

Pre-register: whether the judge's kappa against a human exceeds the two humans' kappa with each other, and whether the judge's error moves pass^k more than pass@k.

My prediction, recorded before you run it: inter-human kappa will be the binding ceiling and lower than people expect, and the judge's bias will distort pass^k by several multiples while barely touching pass@k. Get the opposite and that is a publishable disagreement, not a bug in your setup.

What moved

State of this rung at press time, from a live weekly feed sync across 30 pioneer sources in research-anything (make watch-sync), ranked by technical signal, never audience reach. Every item was fetched from its abstract page on the sync date.

Sync date: 2026-09-11.

• arXiv:2609.11067 (2026-09-10) — When Noise Fabricates Bias. Five noise conditions over 3,822 stereotype-related responses: typos and broken punctuation are far more likely to turn neutral social-bias judgements into biased ones than the reverse, by up to a 120× margin. One day old at press time, and it takes this episode one level deeper — even your measurement of the bias is an instrument with its own error.
• arXiv:2608.29463 (2026-08-29) — Benchmark Contamination: A Taxonomy Organized by Defeated Mitigation. Five contamination types sorted by which mitigation each defeats; a private holdout closes only the first. Two external coders applied the instrument to 41 documents and per-variable weighted kappa ran 0.00 to 0.35 — a paper about measurement honest enough to publish its own poor inter-coder agreement.
• arXiv:2608.26623 (2026-08-27) — AgentJudgeBench. 3,808 instances, six judges from 20B to frontier scale. Alignment degrades with task difficulty, 1.5× faster without ground truth, and on hard queries without ground truth all six converge into a 77–82% band regardless of scale. Structured rubrics buy up to 6.5 points; chain-of-thought and judge temperature buy nothing.
• arXiv:2608.24314 (2026-08-25) — Benchmarking LLM Judges for Voice-Agent Evaluation. The same interactions under three evaluation configurations: judge reliability turns out to be metric- and configuration-dependent rather than a property of the judge. The calibration argument, arriving from telecoms.
• One day outside the window, named anyway: arXiv:2608.11434 (2026-08-11) — MobileJudgeBench, 931 human-annotated trajectories: a simple baseline judge often beats purpose-built pipelines.

[OPEN] What has not moved: I found no published work measuring how much a calibrated judge changes a deployed system's outcomes. Every result above measures the judge against labels, never the consequence of the judge being wrong. That is the gap this episode's Research Challenge points at, and I could not close it from the literature.

Exit test

No scrolling.

1. Name the six rungs in order, and why the sixth was unnamed for five episodes.
2. Your grader agrees with your labels on 79 of 100 items. Why is that not enough, and what number do you need?
3. Which three things is a reported benchmark score a joint fact about, and which one are you usually shown?
4. Why does a fixed-direction judge bias hurt pass^k more than pass@k?
5. An item your judge cannot decide: what should it be recorded as, and what goes wrong in each other option?
6. Name a degenerate input that maximises a metric you report, and the second metric that would destroy it.

Cliffhanger — and the close of the season

You build the golden set and label it yourself, by hand, on a Sunday. Your judge comes back at 0.899, comfortably over the gate. You wire the bench into CI, watch the negative test go red, watch it go green. For one afternoon you believe you have solved the kitchen — which by now is the sound this season makes just before something falls over.

Then someone asks a question your six-rung stack cannot parse: should this agent have been allowed to do that at all?

Not "did it do it correctly" — every rung you own answers that. Whether the action was permitted. By whom, under what authority, recorded where. Your kappa is excellent and has nothing to say, because a calibrated grader tells you whether the answer matched the label and says nothing about whether the label should have been the goal.

That is a seam, not a rung — a next-X question rather than the next episode. Governance is the strongest candidate beyond evaluation: who may change the golden set, who signs off when the gate is overridden, what the audit trail is when an agent acts; then economics, then organisation. Candidates for a future season, in the register Episode 1 used for this one: recorded, argued later, promised never.

The season's real close is smaller. Six rungs, one sentence each, the same shape every time: the prompt is a specification, the context is a budget, the harness is an envelope, the loop is a referee, the graph is a schedule, and the evaluation is the truth function. None is a spell. All six are things you write down — and the last is the only one that can tell you whether the other five were worth writing. Start there next time. It is the rung everybody builds last, and the one everything else was standing on.

Read next

• Prompt Engineering Is a Specification, Not a Spell — Season 7, episode 1 — where the ladder starts, and the {} agent that has now cleared six kitchens
• One Opinion Wearing Ten Hats — a panel is a scoring function, and scoring functions get calibrated or get labelled uncalibrated
• The Rubric That Could Not Fail — I scored my own engine 23 out of 23, and the perfect score was the bug
• Verify Your Verifier — a claim is verified only if its citation resolves and its evidence supports the strength claimed, else it drops to unproven

Episode 6 of Intelligence Engineering Adventures, Season 7 — The Kitchen Ladder, and the close of the season. Claims in the series source are tagged by class — definition, derivation, evidence, engineering choice, open question — and a metaphor may introduce a claim but never serves as evidence for it. Every episode ships a Colab notebook that runs on the free tier with no API key, plus two applications built from nothing — one personal, one enterprise-shaped. The five-rung ladder is Feng et al.'s (arXiv:2608.21156), not mine; the sixth rung is named here, after an external reviewer proposed it on 2026-09-05 and Episode 1 recorded it as an open candidate. The "what moved" section is generated from a live weekly feed sync of 30 pioneer sources, ranked by technical signal and never by audience reach. This article contains no material from any employer or client. — Paul Jialiang Wu · agentic-portfolio-lovat.vercel.app

References

1. Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., Narayanan, A. (2024). AI Agents That Matter. arXiv:2407.01502, submitted 2024-07-01. — quoted verbatim from the arXiv abstract page (read 2026-09-11): the narrow focus on accuracy, the conflated benchmarking needs, and the inadequate holdout sets.
2. Zheng, L., Chiang, W.-L., Sheng, Y. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685, v4 revised 2023-12-24. — quoted verbatim from the arXiv abstract page: the three named biases and the over-80% agreement figure.
3. Yao, S., Shinn, N., Razavi, P., Narasimhan, K. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045, submitted 2024-06-17. — quoted verbatim from the arXiv abstract page: pass^k, and pass^8 under 25% in retail.
4. Liang, P., Bommasani, R., Lee, T. et al. (2022). Holistic Evaluation of Language Models. arXiv:2211.09110, v2 revised 2023-10-01. — abstract read for the 17.9% → 96.0% scenario-coverage figures and the seven metrics.
5. Chang, Y., Wang, X., Wang, J. et al. (2023). A Survey on Evaluation of Large Language Models. arXiv:2307.03109, v9 revised 2023-12-29. — abstract read for the what/where/how framing and the "evaluation as a discipline" argument.
6. Jimenez, C. E., Yang, J., Wettig, A. et al. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv:2310.06770, v3 revised 2024-11-11. — abstract read for the 2,294 tasks and the 1.96% figure.
7. Husain, H. (2024-03-29). Your AI Product Needs Evals. hamel.dev/blog/posts/evals/. — quoted verbatim from the live post (read 2026-09-11): the common root cause of unsuccessful LLM products.
8. Anthropic (2026-01-09). Demystifying evals for AI agents. anthropic.com/engineering. — quoted verbatim from the live post: the definition of an eval, and automated evals in CI/CD.
9. OpenAI. Evals. github.com/openai/evals. — repository read 2026-09-11; described, not quoted.
10. Angulo, J., Yeste, V., Espinos-Morato, H. (2026). Benchmark Contamination: A Taxonomy Organized by Defeated Mitigation. arXiv:2608.29463, submitted 2026-08-29. — quoted verbatim from the arXiv abstract page: leaderboards publish the model and the score.
11. Verma, A., Saha, A. K., Subramanian, S., Aluru, S. H. (2026). AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling. arXiv:2608.26623, submitted 2026-08-27. — abstract read for the 77–82% band and the rubric effect.
12. Purwar, A., Singh, S., Srivastava, K. (2026). Benchmarking LLM Judges for Voice-Agent Evaluation: Reliability, Calibration, and Human Oversight. arXiv:2608.24314, submitted 2026-08-25. — abstract read.
13. Ryu, D., Lee, J., Hwang, Y., Bak, J. (2026). When Noise Fabricates Bias: The Fragility of LLM-as-a-Judge Bias Measurement under Noisy Text. arXiv:2609.11067, submitted 2026-09-10. — abstract read for the 120× asymmetry.
14. Wang, Z., Gu, L., Chi, Z. et al. (2026). Benchmarking LLM Judges for Mobile Agent Evaluation. arXiv:2608.11434, submitted 2026-08-11. — abstract read; dated one day outside this episode's "what moved" window and labelled as such.
15. Goodhart, C. A. E. (1984). Problems of Monetary Management: The UK Experience. In Monetary Theory and Practice, Palgrave, London. link.springer.com. — the 1975 paper reprinted; paywalled, therefore described and not quoted.
16. Strathern, M. (1997). 'Improving ratings': audit in the British University system. European Review 5(3), 305–321. doi:10.1017/S1062798700002660. — the source of the slogan usually attributed to Goodhart; paywalled, therefore described and not quoted.
17. Landis, J. R., Koch, G. G. (1977). The Measurement of Observer Agreement for Categorical Data. Biometrics 33(1), 159–174. doi:10.2307/2529310. — the source of the conventional kappa bands; paywalled, therefore described and not quoted.
18. Feng, Y., Xiang, Z., Yang, C., Ma, Q. (2026). Graph Engineering in the Era of LLM Agents: From Individual Intelligence to System Intelligence. arXiv:2608.21156v2, revised 2026-08-26. — the five-rung ladder this season extends by one.
19. Dhage, S. (2026). Harness Engineering for Predictable Agentic Systems. arXiv:2608.26197, 2026-08-25. — Episode 1's central measurement, cited here for the determinism metric that reads 1.000 on an empty answer.
20. Wu, P. J. (2026-08-06). One Opinion Wearing Ten Hats. agentic-portfolio-lovat.vercel.app. — my own published work: a panel is a scoring function, and a scoring function gets calibrated against something external or gets labelled uncalibrated.
21. Wu, P. J. (2026-08-21). The Rubric That Could Not Fail. agentic-portfolio-lovat.vercel.app. — my own published work: a 23-of-23 score that was the bug, and the hostile criterion that produced an honest 25 of 28.
22. Wu, P. J. (2026-07-23). Verify Your Verifier. agentic-portfolio-lovat.vercel.app. — my own published work: the harness rule this episode's invariant comes from — a claim is verified only if its citation resolves and its evidence supports the strength claimed, else it drops to unproven.
23. Wu, P. J. (2026-09-11). Lab 06 — The truth function: judge drift and pass^k, with the-report-card and the-audit-bench. gist.github.com/wjlgatech/752a9450…. — every number in the Notebook and Projects sections is from these runs, executed 2026-09-11 before publication.