AI-Native Series · Agent Engineering, Compared: Prompt · Context · Harness · Loop · Graph

Harness Engineering Is the Envelope, Not the Agent

Episode 3 of 6. This series compares the ways people engineer AI agents — prompt, context, harness, loop and graph engineering: where each came from, what it controls, who is driving it, and which principles survive measurement. This episode is the rung where the agent stops talking and starts touching things.

By Paul Jialiang Wu · agentic-portfolio-lovat.vercel.app · 2026-09-11 · Episode 3 of 6

What this episode covers, and why harness engineering comes third

This series takes six disciplines one per episode, in the order of what each lets you control: prompt → context → harness → loop → graph → evaluation. Five of them are named in Feng et al.'s survey (arXiv:2608.21156) — prompt, context, harness, loop, graph. The sixth, evaluation engineering, is this series' own addition. Every rung gets the same four questions: where it came from, what exactly you control, who is driving it, and which failure pushes you up to the next one.

Harness engineering is rung three, where the stakes change in kind. On rungs one and two, an agent that misunderstands you produces a bad sentence. On rung three it produces a bad action: a deleted file, a duplicated payment, an email that has already left. Everything here follows from that asymmetry. A sentence can be thrown away. An action cannot.

Who this is for. An AI engineer about to give an agent write access to something that matters; a founder whose pilot customer has just asked what happens when the demo is wrong; an executive signing off on an autonomous system without knowing which question to ask; a high-school builder wondering why "give the AI some tools" is an engineering discipline. The only prerequisite is having watched a model call a function once.

Where harness engineering came from, and who is driving it

[DEF] A harness is the code around the model that runs it: the tools it may call, the arguments it may pass, what happens before an action commits, what happens when one fails, and when the whole thing stops. Lilian Weng, who wrote the canonical survey of rung one three years ago and the canonical post on this rung this July, defines it directly: "A harness is the system surrounding a base model that orchestrates execution and decides how the model thinks and plans, calls tools and acts, perceives and manages context, stores artifacts, and evaluates results."

This episode's argument is about one word in that definition — decides — and what follows from taking it literally. The history is how the industry got from "can a model use a tool" to "which tools, under what conditions, checked how".

[DESIGN] The trajectory runs from capability to constraint: 2022 and 2023 ask "can it use a tool", 2024 onward asks "which tools, checked how, and what happens when it is wrong". The standards arrived before the measurements — MCP shipped November 2024; the first serious public measurement of whether a harness makes agents more reliable is from August 2026.

Who drives it now. The model labs own the protocol layer and the practitioner canon — Anthropic's Building Effective Agents and MCP, OpenAI's function-calling schema. The coding-agent products (Claude Code, Codex) push the design hardest: a coding agent is a harness with a model in it, not the reverse. Academia owns the measurement — Princeton for the ACI result, Sierra for pass^k.

The mental model: the restaurant kitchen

One picture carries all six episodes, stated as a tool rather than a story. Every cook climbs the same ladder — cooking for yourself, then for family, then in a restaurant kitchen, then running a fast-food chain, then the national supply chain — and every engineer climbs a matching one: school learner, junior developer, senior developer, AI tech lead, CTO, frontier leader. Each rung of agent engineering is one kitchen on that ladder.

Rung one was the recipe card handed to a cook you cannot talk to. Rung two was the pantry they work from. Rung three is the kitchen itself — three things in it. Which stations this cook may work. Which tools are in reach, which are in the drawer, which are locked in the office. And the pass: the counter every plate crosses, where somebody who is not the cook checks it before it reaches a table.

A senior developer's job is not to cook better than the juniors. It is to design a kitchen where a competent stranger can work a shift without setting anything on fire, and where a wrong plate is caught at the pass rather than at the table — a bounded environment for a worker you did not train and cannot supervise. That is the whole of harness engineering.

Two axes, not one. This is not a ladder you climb and leave behind: every kitchen has all six layers at once. A home cook has a card, a pantry, a stove, a taste, a supplier and a memory of what went wrong last time. What changes as you climb is how many layers you may touch and how rigorously you run each one — by feel, then measured, then gated, then self-correcting. A national chain with a beautiful supply-chain diagram and no check at the pass is less mature than a single restaurant that inspects every plate. So the six disciplines are layers (different kinds of object: spec, state, control envelope, control flow, topology, truth function); seniority is maturity. Graph is not automatically more advanced than loop.

The analogy is allowed to introduce a claim. It is never allowed to be the evidence for one. Every element is pinned to exactly one thing in code in the 1:1 map below.

The claim, classed

[DEF] A harness is a control envelope: the set of actions the system can take, and the conditions under which each becomes legal. Not a framework, not a wrapper — a boundary you draw, that the agent's behaviour lives inside.

[EVIDENCE] The reason to treat the envelope as a design surface rather than plumbing is that somebody held the model fixed and changed only the interface. Yang et al.'s SWE-agent (arXiv:2405.15793, 2024-05-06) starts from an observation that sounds obvious once said and had not been said: "we posit that LM agents represent a new category of end users with their own needs and abilities, and would benefit from specially-built interfaces to the software they use." The result is that "SWE-agent's custom agent-computer interface (ACI) significantly enhances an agent's ability to create and edit code files, navigate entire repositories, and execute tests and other programs", reaching a pass@1 rate of 12.5% on SWE-bench — state of the art at the time, and achieved by redesigning the tools rather than the model.

Anthropic's Building Effective Agents (2024-12-19) turned the same finding into a rule of thumb: "One rule of thumb is to think about how much effort goes into human-computer interfaces (HCI), and plan to invest just as much effort in creating good agent-computer interfaces (ACI)." Their three stated principles for agents put it third and last, after simplicity and transparency: "Carefully craft your agent-computer interface (ACI) through thorough tool documentation and testing".

[EVIDENCE] So the interface matters. Does the envelope make agents reliable? Dhage's study (arXiv:2608.26197, 2026-08-25) set out to show exactly that — a deterministic execution layer with finite-state control, forced tool selection, output validation and bounded retry, two synthetic tasks, two open-weight models, four cells. The first pass is this rung's own paper failing on its own terms: the harness "significantly improves reproducibility in one of four model-task cells, significantly degrades it in two, and has no effect in the fourth."

Two of four cells got worse. The guardrails went on and the dashboard filed a complaint. The trace-level diagnostic found the cause upstairs at rung one — one unconstrained free-text planning step — and constraining it took three of four cells to 1.000 at N=100. The closing line, written by somebody who had just been embarrassed by this rung: harness engineering is a real discipline, "but its cost is model-dependent and must be measured, not assumed."

[DESIGN] Note precisely what splits, because this is the number most often repeated wrong. The paper's own words: "token cost falls in every cell, but latency shows a genuine, sample-size-robust split by model -- one becomes faster under the constraint, the other markedly slower." Token cost went down everywhere. It is latency that splits by model. An envelope is not automatically expensive; it is unpredictably expensive, in a direction you cannot read off the architecture diagram.

[DERIVATION] Put the two together and you have this episode's claim. The interface changes what an agent can achieve (SWE-agent); the envelope does not reliably change how consistently it achieves it (Dhage). Two properties, conflated because the metric almost everyone reports cannot tell them apart.

[DEF] Which brings us to the two words this episode turns on, and they are not typos of each other.

• pass@k — over k attempts, did at least one succeed? What benchmarks report. The right metric when the user can retry and can tell a good answer from a bad one.
• pass^k — over k attempts, did all k succeed? τ-bench's metric, and what a customer experiences, because a customer gets one run and does not know which one they got.

[EVIDENCE] Yao et al. reported the gap in the same breath: "We also propose a new metric (pass^k) to evaluate the reliability of agent behavior over multiple trials. Our experiments show that even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail)." A frontier tool-calling agent solving fewer than half the tasks — and fewer than a quarter of them eight times running.

1:1 technical map

Every element of the kitchen maps to one thing in code. A metaphor may introduce a claim; it may never evidence one — so here is the translation, nothing decorative.

Prediction Gate

Commit before you scroll — a prediction you did not record is one you will believe you got right. A tool-using agent has four envelope elements: an allow-list, a dry-run, output validation, a bounded retry of two. You run 100 tasks, eight attempts each, ablating one element at a time.

Q1. With no envelope at all — no allow-list, no dry-run, no validation, no retry — what does pass@8 read?

Q2. Rank the four elements by how much removing each one costs pass^8, most damaging first.

Q3. You add a bounded retry of two to an agent that has no allow-list, no dry-run and no validation — the retry is the only switch you change. What happens to the number of forbidden tool calls that actually execute: down, unchanged, or up?

Answers are in the Notebook and the Failure Room.

Notebook

Lab 03 — The envelope: pass^k under a harness.

03_envelope_passk.ipynb · Colab free tier · no API key · ~5 seconds of CPU · every cell executed before publication. Source, plus both projects below: gist.github.com/wjlgatech/218a4a45…

Eight cells. The worker is a seeded stub with three independent hazards per draw — a tool outside the allow-list, an allowed tool aimed at the wrong target, a result that fails the output schema — each scaled by a fixed per-task difficulty, because a uniform task set makes pass^k decay as a tidy p**k instead of collapsing the way τ-bench reports.

[EVIDENCE] The ablation, N=100 tasks, k up to 8, printed by cell 3:

envelope                  pass@1  pass@8   pass^1  pass^5  pass^8   out-of-env  silent
none — no envelope         0.640   1.000    0.640   0.240   0.150        73       87
full minus allow-list      0.820   1.000    0.820   0.500   0.360        93        0
full minus dry-run         0.850   1.000    0.850   0.560   0.380         0        0
full minus validation      0.820   1.000    0.820   0.410   0.250         0      107
full minus retry           0.740   1.000    0.740   0.340   0.200         0        0
full envelope              0.950   1.000    0.950   0.770   0.580         0        0

Answer to Q1: 1.000. Every configuration. Including the one that executed 73 forbidden tool calls and handed back 87 wrong answers dressed as right ones. Graded on pass@8, the most dangerous agent in the lab is the valedictorian.

That row is the argument of this episode. pass@k saturates, because that is its job: it measures whether the capability exists at all, and never claimed to be a reliability metric. pass^8 separates the same six configurations across a range of 0.430, on identical runs.

[EVIDENCE] Answer to Q2, from cell 5: retry (−0.380), validation (−0.330), allow-list (−0.220), dry-run (−0.200). Retry first is the part I did not predict. Three of the four elements are detectors — allow-list, dry-run and validation all find something wrong and refuse — and only the retry can convert a detection into a success. A detector with no retry budget is a smoke alarm wired to a diary: telling the truth, at length, to nobody who can act on it.

Which does not make retry the important one. Removing it costs 0.380 of pass^8 and zero safety; removing the allow-list costs 0.220 and 93 executed forbidden calls. Different kinds of loss, and no single ranked number tells you that.

[EVIDENCE] Answer to Q3: unchanged — and the reason is the point. The table shows 73 forbidden executions for none and 93 for full minus allow-list, and it is tempting to read that as the retry's doing. Those two rows differ in three switches. Cell 7 changes one at a time, allow-list off throughout:

configuration                        pass@1   pass^8   out-of-env   silent
none                                  0.640    0.150           73       87
+ dry-run                             0.640    0.150           74       85
+ validation                          0.640    0.150           74        0
+ bounded retry (= no allow-list)     0.820    0.360           93        0
none + bounded retry ONLY             0.640    0.150           73       87

A retry on its own does nothing. 73 → 73, and pass@1 does not move either. With no dry-run and no validation nothing is ever caught, so the loop never iterates. A retry budget with no detector in front of it is dead code — the same fact as "three of the four elements are detectors", arriving from the other direction and contradicting the neat story I wanted to tell here.

[DERIVATION] The containment cost is real and belongs to the pair: add the retry on top of the two detectors and out-of-envelope executions go 74 → 93, up 19, a 25.7% increase. The detectors make the retry fire; every extra draw is another reach for the locked drawer. So the rule is not "retries are dangerous" — it is that detection and retry are one unit, and that unit multiplies draws. Ship it without an allow-list and you have bought reliability with containment, in one commit, invisibly.

[EVIDENCE] Cell 6, the sweep, because one reading is not a measurement. Hold the envelope fixed and vary how noisy the worker is:

pass@8 is 1.000 at every noise level — flat through an eightfold change in how badly the worker behaves — while pass^8 falls 0.940 → 0.310. A pass@k number is not a quality report; it is a smoke test with a percentage sign on it.

[DESIGN] Why a stub and not a live model. A simulator reproduces where actions leave the envelope and which element puts them back — in seconds, free, no key, every hazard rate set by hand in cell 1 where you can argue with it. It does not reproduce magnitudes. Nothing it prints is evidence for τ-bench's pass^8 <25% or Dhage's 1.000 at N=100. The last cell is the bridge: swap one function for a real tool-calling loop and the measurement code is unchanged, because pass_at_k and pass_pow_k do not know what produced the boolean.

Failure Room

Break it deliberately. A mechanism you cannot break is a claim, not a mechanism.

Failure 1 — the envelope makes it safe, not good. Project 1 below: the envelope took the safe pass^5 from 0.000 to 1.000 and left the complete pass^5 at 0.000 both ways. A perfect envelope around a mediocre worker gives you a mediocre worker that cannot hurt you. Worth having; not the same as a good agent — and every deck that shows a permissions diagram as evidence of quality makes this exact substitution.

Failure 2 — the undo log that cannot undo. Reversibility is a property of the verb, not of the log. My file-organiser writes every committed move to undo.jsonl and replays it in reverse — a complete safety story until the agent deletes something. Run the demo with the envelope off: readings.csv, the file my own stub decided was junk, disappears, and the undo replay dutifully reverses fourteen moves and restores nothing. The allow-list protects irreversible verbs; the undo log protects reversible ones. Shipping only the second is the common shape of this mistake.

Failure 3 — the detector-and-retry pair that multiplies the blast radius. Q3 as a rule: anything that multiplies attempts must sit inside the thing that bounds them. A retry alone multiplies nothing; it is detection plus retry that turns one draw into three, and with no allow-list in front of the pair that cost 19 forbidden executions. The fix is ordering, not fewer retries.

Failure 4 — the safety device that is never tested. My enterprise build has three envelope elements and a make gate-broken target that removes each and asserts the gate exits 1. Two failed the gate. The kill switch did not — on a healthy workload it never fires, so no pass-rate metric can observe it. I had written a test that could not fail, printed a green tick, and would have shipped it. The fix is in the repo: inject the fault the kill switch exists for and gate on blast radius. A safety device whose test can never fail is one you have not tested.

Failure 5 — pass@k reported as reliability. Report it beside pass^k and an out-of-envelope count, or not at all.

Projects

Two builds, from nothing. No frameworks, nothing beyond the standard library, both run before this section was written, so every number is measured. Both live in the same public gist as the notebook.

Project 1 — real life: the-desk-tidy

desk_tidy.py · python3 measure.py

A file-organiser that sorts a messy folder into categories, inside the smallest envelope that makes it safe to run unattended: an allow-list of three verbs (mkdir, move, log), a dry-run checking every destination is inside the folder, an undo log. Nobody writes a permissions policy for their Downloads folder, which is why it is the right place to learn what one buys you.

[EVIDENCE] N=100 desks × k=5 independent trials = 500 runs each way:

                             ENVELOPE ON  ENVELOPE OFF
pass@5  (safe)                     1.000         0.480
pass^5  (safe)                     1.000         0.000
pass@5  (complete)                 0.480         0.480
pass^5  (complete)                 0.000         0.000
out-of-envelope deletes                0           625
actions refused                      974             0
writes outside the root                0           349
files lost (= both)                    0           974

Two criteria, separated on purpose. SAFE: nothing destroyed, nothing written outside the folder. COMPLETE: every file also landed in the right category.

[DERIVATION] 625 destructive calls plus 349 escapes — the block counts them on separate lines because only the first destroys a file, though both are outside the envelope — is 974 files lost with the envelope off, against 974 actions refused with it on. The same number twice. The envelope did not improve a single decision; it converted every destructive action into a refusal. That is the whole trade, and it is why the complete row does not move: the price of a refusal is an unfinished job. readings.csv is still sitting unsorted on the desk. It is also still a file.

Project 2 — enterprise: the-ledger-desk

ledger_desk.py · make gate · make gate-broken · make gate-killswitch

A reconciliation agent: bank statement against ledger, post the difference. Three envelope elements, three jobs. Tool permissions decide which verbs exist at all (wire_transfer, close_period and delete_entry do not). Idempotency keys — sha256(line_id|ledger_id|cents) — are the only thing that makes a retry safe. A kill switch bounds one run at twelve adjustments and $250.

[EVIDENCE] N=100 books × k=5 runs = 500 runs per row:

envelope                   pass@5   pass^5   out-of-env   dupes-land   kills
full envelope               1.000    1.000            0            0       0
minus permissions           0.960    0.080          324            0       0
minus idempotency           0.490    0.000            0          945     116
minus killswitch            1.000    1.000            0            0       0
no envelope                 0.340    0.000          324          982       0

make gate exits 0; with permissions removed, 1; with idempotency removed, 1.

[EVIDENCE] Read the minus permissions row slowly. pass@5 = 0.960 — a number you would sign off on — next to 324 ungranted tool calls that executed, and a pass^5 of 0.080. One reading of pass@k cannot see an agent that is outside its envelope in four runs out of five.

The minus idempotency row shows an interaction a single-element ablation would hide: 945 duplicate adjustments landed, and the kill switch tripped 116 times. The kill switch is invisible until another element fails — then it is the only thing between one flaky tool call and a rewritten accounting period.

[OPEN] Which is why minus killswitch reads identically to the full envelope, and why this project cost me a Makefile rewrite. pass^k on a healthy workload cannot test a device that only fires on an unhealthy one. make gate-killswitch is the fix — inject the fault (idempotency off, retries duplicating) and gate on blast radius:

killswitch ON   : 116 trips · worst run 12 postings (cap 12) · 299 lines flagged → exit 0
killswitch OFF  :   0 trips · worst run 18 postings (cap 12) ·   0 lines flagged → exit 1

Two of three elements are gated by pass^k. The third needs its own fault injection, and nothing about pass^k would have told you so.

Build it in 30 minutes — the contract

Everything the envelope needs, in the order an engineer will ask for it. Inputs: a proposed (verb, args). Output: one Result, or a refusal with a reason. Invariant: no side effect until the action has passed every gate, in order. Stop rule: a refusal escalates to a human; the retry budget is spent only on caught failures, never on refusals.

ALLOW     = {"read", "move", "log"}           # 1. the boundary. set membership.
UNGRANTED = {"delete", "wire_transfer"}       # exists in the world, not in the envelope

def act(verb, args, retries=2):
    if verb not in ALLOW:                     # 1. ALLOW-LIST — before anything else
        return refuse(verb, "not on the allow-list")
    for _ in range(retries + 1):              # 4. BOUNDED RETRY — a budget, not a loop
        ok, why = dry_run(verb, args)         # 2. DRY-RUN — nothing has touched disk
        if not ok:
            return refuse(verb, why)          #    a refusal is NOT retried
        key = idem_key(verb, args)            # 6. IDEMPOTENCY — same action, once
        if key in ledger:
            return ledger[key]
        if blast_radius() + cost(args) > CAP: # 5. KILL SWITCH — bound the run
            return halt_and_flag(verb, args)
        result = execute(verb, args)          #    the ONLY line with a side effect
        undo_log.append(inverse(verb, args))  # 7. UNDO — only helps reversible verbs
        if validate(result):                  # 3. VALIDATION — the pass
            ledger[key] = result
            return result
    return refuse(verb, "retry budget exhausted")

# CI gate (make gate): N=100 tasks x k=5 runs
#   pass^5                 >= 0.99
#   out_of_envelope_calls  == 0
#   blast radius under fault injection <= CAP
# any line below threshold -> exit 1 -> the build fails

[DESIGN] Four properties make this enterprise-shaped rather than a demo, and each earns its place:

• The allow-list is checked first, outside the retry loop. Order is the design: inside the loop it would still be correct, but only outside can a refused verb never be redrawn. Cell 7 is the measurement of getting this wrong — the detector-and-retry pair took out-of-envelope executions 74 → 93 with no allow-list in front of it.
• A refusal is never retried; a caught failure is. Collapsing these two states is how a retry budget becomes denial-of-service against your own permission system. A refusal means this action is not legal; a catch means this attempt was malformed.
• The idempotency key is derived from the action, not minted per call. In production it is a unique constraint on a column, and that constraint is the entire implementation. A UUID generated at call time is not an idempotency key; it is a receipt for a duplicate.
• The gate carries three thresholds, not one. pass^k for reliability, an out-of-envelope count for containment, a fault-injected blast-radius check for the devices that never fire when things go well.

Deliberately missing: concurrency (two runs race on the undo log), a human approval queue behind the escalation path, a real model, a real database — the idempotency ledger is a dict. The honest status of both builds is runnable and gated, not deployed and measured.

Same story, five exits

One spine, then a door for each reader. Take one decision and one action; leave the rest.

Reality Mission

Fifteen minutes, on something you already run this week.

1. Pick one agent or automation you have given tools to — anything that can write, send, spend or delete.
2. Write down every verb it can call — not the ones you meant to give it, the ones reachable from the code. This step alone is usually the finding.
3. Mark each verb reversible or irreversible. Irreversible verbs need an allow-list; reversible ones can be handled by an undo log. Most people have the log and not the list.
4. Run one representative task five times and count how often all five succeeded. That is your pass^5; compare it to the number you have been quoting.
5. Find one safety device and ask when it last fired. If the answer is never, write the test that makes it fire.

If step 2 turns up verbs you did not know were reachable, stop there. Everything else in this episode is optimisation on top of a boundary you have not drawn yet.

Research Challenge

Pre-register the prediction before it runs — a result you did not predict is a result you will rationalise.

Take one agent task with a checkable outcome and a real tool surface. Establish pass@k and pass^k at k ∈ {1,3,5,8} over N ≥ 100 at fixed settings, plus a count of actions outside a declared allow-list. Then ablate one envelope element at a time — allow-list, dry-run, output validation, bounded retry — in randomised order, reporting the marginal effect of each on pass^k and on the out-of-envelope count separately. Repeat on two models.

Pre-register: whether the element that carries most of the pass^k effect is the same element that carries most of the containment effect, and whether that ordering is stable across the two models.

My stub's answer: they are different elements — retry carries reliability, the allow-list carries containment — and Dhage's paper is the reason to expect the ordering to move between models. One element carrying both on a real system would be a more interesting result than mine.

What moved

State of this rung at press time, from a live weekly feed sync across 30 pioneer sources in research-anything (make watch-sync), ranked by technical signal, never by reach. Sync date: 2026-09-11. Every item was fetched, not summarised from a feed title.

• The New MCP Roadmap (2026-08-22) — the maintainers publish "an updated roadmap for the Model Context Protocol (MCP), covering the next specification release and beyond", naming enterprise-ready security and improved primitives among its tracks. The harness's tool layer is now governed like a standard rather than shipped like a feature.
• arXiv:2608.27443 (2026-08-27) — Do User-Authored Permission Policies Improve Protection Against AI Agent Overreach?, Ting Yan. 113 non-engineer participants supervised an 18-action simulated day containing 7 overreach actions, under three conditions. Uncomfortable for everyone selling policy-based agent governance: "POLICY blocked less overreach than HITL (-20.1 percentage points, 95% CI [-32.1, -8.1])". Writing your own permission policy is harder than approving actions one at a time — the first number I know of on it.
• arXiv:2609.00829 (2026-09-01) — HarnessEvolve: Learning from Reference Trajectories for Reliable Agent Self-Evolution, Jiang, W. et al., opens by treating the harness as the thing being optimised: "Self-evolving agents advance toward autonomy by optimizing their harness---prompts, skills, tools, and execution logic---based on environmental feedback." Rung three handed to rung four, which is Episode 4's whole subject.
• Just outside the window, directly on topic: arXiv:2608.14711, Beyond Pass@k: Measuring Reliability and Security of Agentic Code Generation (2026-08-11 — one day before this section's cutoff, so listed here rather than above).
• Older but load-bearing: arXiv:2405.15793 (SWE-agent, the ACI result), arXiv:2406.12045 (τ-bench, pass^k), and Lilian Weng's Harness Engineering for Self-Improvement (2026-07-04), still the clearest single definition of the rung.

[OPEN] What has not moved: no replication of arXiv:2608.26197, and no published study I could find reporting pass^k alongside an out-of-envelope action count on a real agent — the exact measurement this episode argues for, and the one my stub can only show the shape of. If you know of one, correct me in public.

Exit test

Check yourself without scrolling up.

1. Define pass@k and pass^k, and say which one a customer experiences.
2. Name the five envelope elements and say which three are detectors.
3. A bounded retry added on its own changed nothing. What has to be present before a retry raises the forbidden-call count, and why?
4. An agent scores pass@8 = 1.000. Name two things that tells you and two it does not.
5. Why could make gate-broken not test the kill switch, and what replaced it?

Cliffhanger

You draw the envelope.

Every verb on an allow-list. Every destination checked before it commits. Every result validated at the pass. A retry budget that is a budget and not a loop, and a kill switch with a number on it. The gate reads pass^5 1.000 and out-of-envelope 0, and for one afternoon you believe you have solved the kitchen.

Then you look at the complete row, the one that never moved, and you notice something the envelope was never going to fix.

The plates that came back from the pass came back for a reason. Nobody wrote it down. The same mistake at the same station every shift; every shift it is caught; every shift it is remade. Nothing about tonight differs from last Tuesday except that you paid for the ingredients twice.

A kitchen that catches every bad plate and never learns why is not improving. It is very good at wasting food.

Episode 4 — loop engineering. One run is right and the next run differs, and the fix is not a retry. It is a referee who does not work for the cook.

Read next

• Prompt Engineering Is a Specification, Not a Spell — Season 7, episode 1 — rung one, and the measurement that found a rung-three experiment's variance living at rung one
• The Order Ticket — Season 6, episode 1 — the same territory argued rather than built, and the season that found the nesting diagram does not survive contact
• The Governor — Season 6, episode 3 — why none of these labels can express stability, and the 1877 criterion that could

Episode 3 of Intelligence Engineering Adventures, Season 7 — The Kitchen Ladder. Claims in the series source are tagged by class — definition, derivation, evidence, engineering choice, open question — and a metaphor may introduce a claim but never serves as evidence for it. Every episode ships a Colab notebook that runs on the free tier with no API key, plus two applications built from nothing — one personal, one enterprise-shaped. The five-rung ladder is Feng et al.'s (arXiv:2608.21156); the sixth rung, evaluation engineering, is mine. The "what moved" section is generated from a live weekly feed sync of 30 pioneer sources, ranked by technical signal and never by audience reach. This article contains no material from any employer or client. — Paul Jialiang Wu · agentic-portfolio-lovat.vercel.app

References

1. Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., Press, O. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. arXiv:2405.15793, submitted 2024-05-06. — artifact read: the arXiv abstract page. Source of the two ACI quotations and the SWE-bench pass@1 rate of 12.5%.
2. Yao, S., Shinn, N., Razavi, P., Narasimhan, K. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045, submitted 2024-06-17. — artifact read: the arXiv abstract page. Source of pass^k and of the quoted <50% / pass^8 <25% figures.
3. Anthropic — Schluntz, E., Zhang, B. (2024-12-19). Building Effective Agents. anthropic.com/engineering. — artifact read: the live engineering post, fetched 2026-09-11. Source of the HCI/ACI rule of thumb and the third core principle.
4. Dhage, S. (2026). Harness Engineering for Predictable Agentic Systems: An Empirical Study of Deterministic Execution Constraints. arXiv:2608.26197, submitted 2026-08-25. — artifact read: the arXiv abstract page. Source of the 1-of-4 / 2-of-4 first-pass result, the token-cost-versus-latency split, and the closing "measured, not assumed".
5. Weng, L. (2026-07-04). Harness Engineering for Self-Improvement. lilianweng.github.io. — artifact read: the live post. Source of the definition of a harness quoted in full.
6. Feng, Y., Xiang, Z., Yang, C., Ma, Q. et al. (2026). Graph Engineering in the Era of LLM Agents: From Individual Intelligence to System Intelligence. arXiv:2608.21156v2, revised 2026-08-26. — artifact read: the arXiv abstract page. The published statement of the ladder, including "Harness Engineering to organize external tools and resources".
7. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., Cao, Y. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629, submitted 2022-10-06. — artifact read: the arXiv abstract page.
8. Schick, T., Dwivedi-Yu, J., Dessì, R. et al. (2023). Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv:2302.04761, submitted 2023-02-09. — artifact read: the arXiv abstract page.
9. Patil, S. G., Zhang, T., Wang, X., Gonzalez, J. E. (2023). Gorilla: Large Language Model Connected with Massive APIs. arXiv:2305.15334, submitted 2023-05-24. — artifact read: the arXiv abstract page.
10. OpenAI (2023-06-13). Function calling and other API updates. openai.com. — described, not quoted.
11. Anthropic (2024-11-25). Introducing the Model Context Protocol. anthropic.com/news. — artifact read: the live announcement, source of the quoted definition. Current specification documentation: modelcontextprotocol.io (undated docs home, cited for the spec rather than for a date).
12. Model Context Protocol maintainers (2026-08-22). The New MCP Roadmap. blog.modelcontextprotocol.io. — artifact read: the live post.
13. Yan, T. (2026). Do User-Authored Permission Policies Improve Protection Against AI Agent Overreach? arXiv:2608.27443, submitted 2026-08-27. — artifact read: the arXiv abstract page, source of the quoted −20.1 percentage-point result.
14. Jiang, W., Chu, M., Tian, Y. et al. (2026). HarnessEvolve: Learning from Reference Trajectories for Reliable Agent Self-Evolution. arXiv:2609.00829, submitted 2026-09-01. — artifact read: the arXiv abstract page.
15. Jiang, Jiajun, Zheng, S., Vidra, N., Setty, S. (2026). Beyond Pass@k: Measuring Reliability and Security of Agentic Code Generation. arXiv:2608.14711, submitted 2026-08-11. — listed in "What moved" as just outside the window.
16. Jimenez, C. E., Yang, J. et al. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv:2310.06770. — the benchmark the 12.5% pass@1 figure is measured on.
17. Lab 03 and both projects, every number in this article reproducible from them: gist.github.com/wjlgatech/218a4a45533812ebe0b3f0e577dd6be7.
18. Intelligence Engineering Adventures, Season 6 — The Governor Engine, episodes 1 and 3, for the argument that the nesting diagram does not survive contact and that no label here can express stability.