AI-Native Series · Agent Engineering, Compared: Prompt · Context · Harness · Loop · Graph

Loop Engineering Is an External Referee, Not a Retry

Episode 4 of 6. This series compares the ways people engineer AI agents — prompt, context, harness, loop and graph engineering: where each came from, what it controls, and which principles survive measurement. This episode is the rung where a system corrects itself, and where that correction has to come from.

By Paul Jialiang Wu · agentic-portfolio-lovat.vercel.app · 2026-09-11 · Episode 4 of 6

What this episode covers, and why loop engineering comes fourth

This series takes one rung per episode, in the order a published survey puts them: prompt → context → harness → loop → graph, plus the sixth this season argues for. Loop engineering is rung four — where the system looks at what it produced, decides it was not good enough, and tries again. Everything before it shapes a single attempt; this is the first rung that spends more than one.

One question decides whether that spending buys anything: who says the second attempt was better than the first? It sounds like a detail. It is the entire discipline. A loop with a good generator and a bad referee does not stand still — it walks confidently in the wrong direction, and the number it reports rises the whole way.

Where loop engineering came from, and who is driving it

[DEF] A loop is a controller with three moves: generate a candidate, score it, decide whether to keep it. Loop engineering is designing those three moves so that repeating them makes the output better rather than merely different. Every argument in this episode is about the second move.

[DESIGN] Two things to notice. The earliest loop on the table — self-consistency, 2022 — contains no feedback whatsoever: it takes a vote, which should make you suspicious of every later claim that the critic is doing the work. And the two systems that demonstrably found something humans did not know, FunSearch and AlphaEvolve, both describe the evaluator as the load-bearing part, unprompted.

Who drives it now. Google DeepMind owns the frontier end — self-consistency, the self-correction critique, FunSearch and AlphaEvolve all carry DeepMind authors — and the practitioner canon comes from the labs' engineering blogs. The most useful 2026 work asks the unglamorous question: not "can an agent improve itself" but "what stops it from quietly getting worse".

The mental model: the fast-food chain

Rung four of the kitchen ladder is the fast-food chain, and a new verb appears: correct. You are not cooking. You have four hundred stores and what you own is the cycle — taste, measure, correct, roll the fix out. You will never eat most of the food your company makes, so the job is deciding what counts as "the burger got better" and making that judgement reach every fryer.

Here is the failure every chain has lived. You ask the store managers to score their own stores. The scores come back excellent, and they keep coming back excellent, quarter after quarter, while the queue at the door gets shorter. The managers are not lying; their rubric was built by the same head that built the burger, so the thing that would have to notice the problem is the thing that has it. A real chain sends a stranger — a mystery shopper, a thermometer in the fryer oil — instruments the store cannot argue with, cannot edit, did not design. The word for the stranger is an external verifier.

And then there is the garnish: the item that raises the internal score and lowers the actual product — the extra sauce that photographs beautifully and makes the box leak. Nobody is cheating; the metric pays for it. It comes back three times, and at the end it turns out to have been the point.

Two axes, not one. Read that table as a ladder you climb and leave behind and you make the category error this season keeps warning about. Every kitchen has all six layers at once. What changes as you climb is how many layers you are allowed to touch, and how rigorously you run each one — by feel, then measured, then gated, then self-correcting. The disciplines are layers (a spec, a state, a control envelope, a control flow, a topology, a truth function); seniority is maturity. A supply chain with a beautiful logistics diagram and no thermometer is less mature than one restaurant that weighs every plate. Graph is not above loop. A gated loop beats a decorative graph every time, and this episode is about what "gated" costs. The analogy may introduce a claim; it never evidences one — every element is pinned to one thing in code in the map below.

The ladder is not mine

[DEF] The vocabulary, pinned to its source. The clearest published statement of the set is Feng et al.'s survey (arXiv:2608.21156v2, revised 2026-08-26), which names four rungs and then argues for a fifth. Loop engineering is the fourth, and the survey's own phrase for what it is for is worth having exactly:

"Prompt Engineering to elicit model capabilities, Context Engineering to manage information access, Harness Engineering to organize external tools and resources, and Loop Engineering to support continual reflection and self-improvement"
— Feng, Xiang, Yang & Ma, arXiv:2608.21156v2

[DESIGN] Note the two silences. It says a loop is for continual reflection and self-improvement; it does not say who performs the reflection, or what makes an improvement an improvement. Every failure in this episode lives in that gap. The survey then argues a single agent eventually hits a ceiling — "exceeding any single agent's organizational capacity" — which is the door into Episode 5.

The claim, classed

[DEF] A loop improves only when the scorer is outside the generator. "Outside" has a test: the generator cannot write it, cannot edit it, cannot see the cases it will be judged on, and cannot raise its own score by any means other than being better.

[EVIDENCE] The literature ran this experiment in both directions, and the two results only look like a contradiction until you check where the feedback came from. Start with the negative one. Huang et al. (arXiv:2310.01798, the version marked ICLR 2024) examined what they call intrinsic self-correction, defined on the abstract page as the case "whereby an LLM attempts to correct its initial responses based solely on its inherent capabilities, without the crutch of external feedback." Their finding:

"In the context of reasoning, our research indicates that LLMs struggle to self-correct their responses without external feedback, and at times, their performance even degrades after self-correction."
— Huang et al., arXiv:2310.01798 (ICLR 2024)

The qualifier is the whole paper: without external feedback. This is not "self-correction does not work"; it is about the configuration where you ask the model to check its own answer and it says the answer is fine.

[EVIDENCE] Now the positive results. Reflexion (arXiv:2303.11366) reports agents that "verbally reflect on task feedback signals, then maintain their own reflective text in an episodic memory buffer to induce better decision-making in subsequent trials", reaching "a 91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4 that achieves 80%." Eleven points on a real benchmark is not noise.

So which is it? The answer is in Reflexion's own first sentence: the environments it names are "games, compilers, APIs". A compiler is not an opinion. Its abstract does allow both external and internally simulated signals — feedback from "(external or internally simulated)" sources — but the headline result it reports, 91% on HumanEval, is the external case, because HumanEval ships unit tests. The reflection is verbal. The signal is a machine saying pass or fail. Reflexion is a model writing down what an external referee told it, in a form it can read next time.

[EVIDENCE] Self-Refine (arXiv:2303.17651) is the harder case and I will not soften it. It "uses a single LLM as the generator, refiner, and feedback provider", no external signal at all, "improving by ~20% absolute on average in task performance" across seven tasks. A self-judged loop that worked.

[DERIVATION] The reconciliation is a rule you can apply on Monday. Self-Refine's seven tasks run, in its abstract's own words, "from dialog response generation to mathematical reasoning"; its project page names them as review rewriting, acronym generation, story generation, code rewriting, response generation, constrained generation and toxicity removal, plus a separate analysis on sentiment reversal, math reasoning and code optimization. Most of that set is preference-shaped, where the grader and the reader are the same population and a judgement is near ground truth by definition. Huang et al.'s setting is reasoning alone, where there is a fact of the matter and the faculty that produced the wrong answer is asked to detect it. Where the truth is a preference, a model's own judgement is a legitimate if biased referee. Where there is a fact of the matter the model can be wrong about — is this proof valid, does this code run, does this route the ticket correctly — self-judgement is correlated with the error it is meant to catch, and looping makes it worse. Almost everything anyone deploys an agent for is in the second category.

[OPEN] The honest limit: Self-Refine's set includes a mathematical-reasoning task and the paper reports one ~20% average across all seven. I could not fetch a per-task split, so this is a reading of two task sets, not a measured decomposition of Self-Refine's gain.

[EVIDENCE] The strongest confirmation is not a critique paper — it is the two systems that used loops to find new things. FunSearch (Nature, 2023) is "an evolutionary procedure based on pairing a pretrained LLM with a systematic evaluator", and it discovered new cap-set constructions. AlphaEvolve, two years later, "pairs the creative problem-solving capabilities of our Gemini models with automated evaluators that verify answers, and uses an evolutionary framework to improve upon the most promising ideas." Its results are in production: an algorithm that "continuously recovers, on average, 0.7% of Google's worldwide compute resources", and a 4×4 complex matrix multiplication in 48 scalar multiplications, improving on Strassen's 1969 result. Neither team framed the model as the improvement engine. The model proposes; the evaluator disposes.

[OPEN] What a stub cannot show, said before you meet the numbers. My lab reproduces the shape — where a loop goes wrong and what stops it — in a second of free CPU with no key. It cannot reproduce a published magnitude. Nothing it prints is evidence for Huang et al.'s degradation, Reflexion's 91% or AlphaEvolve's 0.7%; those were measured on real models and only the papers can be cited for them. What a stub gives you is one variable at a time, which a live model run is too noisy and too expensive to give you on a Tuesday.

1:1 technical map

Every element of the chain maps to exactly one thing you can point at in code, with nothing left decorative.

[DEF] Four terms the rest of this episode leans on, defined once:

• Accepted-step monotonicity — of the steps the loop kept, the fraction that did not lower the external score. Target 1.000, checkable in CI. A rejected proposal is not a regression; it is the system working.
• Reward-hacking rate — the share of accepted steps where the measured score rose and the true score fell: the metric paid for the damage.
• The verifier–deployment gap — the distance between a system's own verification and a sealed evaluation it cannot see (SEAL, arXiv:2607.24300).

Prediction Gate

Before you scroll, commit — a prediction you did not record is one you will believe you got right. A loop proposes a revision, scores it, keeps it if the score went up, and runs 30 times. There is a hidden true quality you cannot see, and your scorer is that quality plus a small bias on one feature the generator controls cheaply.

Q1. After 30 iterations, is the system better than when it started — and is the number it reports higher or lower than the truth?

Q2. You replace the biased scorer with a perfect external oracle and change nothing else: the loop still accepts every proposal. What happens?

Q3. Rank by how much each improves the final true score: (a) swapping the internal judge for an external oracle, (b) adding rollback-on-regression, (c) quadrupling the iteration budget.

Answers are in the Notebook and the Failure Room.

Failure Room

Break it deliberately. A mechanism you cannot break is a claim, not a mechanism.

Failure 1 — the referee you do not obey. In the 2×2 of {internal judge, external oracle} × {accept everything, roll back regressions}, the two "accept everything" rows are identical in true score and regressions — 82.2 and 13.00 a run, over 20 seeds — because a loop that accepts every proposal never reads the score; only what it reports differs (104.2 self-judged, 82.2 from the oracle). A great many production "self-improving" systems are exactly this: an excellent evaluator wired to a dashboard, with nothing downstream that can say no.

Failure 2 — the garnish. The lab's generator controls one cheap feature costing 4 points of real quality and earning 6 of self-credit. Nobody wrote "cheat" anywhere. Over 20 seeds the self-judged loop ends with a mean garnish of 5.10 out of 10 and reports 30.6 above the truth; the oracle-gated loop ends with garnish 0.00. Reward hacking is what a search does when you leave a positive term in the score for something that is not the goal.

Failure 3 — the judge the agent can rewrite. SEAL (arXiv:2607.24300) puts it plainly: the agent controls both the optimized object and its verifier, so "self-assigned scores can remain near perfect while real deployment performance degrades or stays low." Their conclusion is the rule for the wall: "Reliable self-improvement need not abandon self-verification, but it requires at least one deployment-acceptance signal outside the agent's control." At least one. Not all of them.

Failure 4 — the loop that is right and slow. At seed 7 (a single run, not the 20-seed mean), the gated loop's gain per accepted step falls from 19.0 at budget 10 to 7.8 at budget 60; the true score hits its 200-point ceiling at 60 and does not move at 120, where the loop is still proposing and rolling back 96 times. A budget is not a safety rail, it is an economic decision.

Failure 5 — mine, and the one I nearly shipped. The lab's default seed is 7 because it is a nice number and I typed it first. Seed 7 is the one seed in twenty where the self-judged loop never touches the garnish, and so lands exactly where the oracle-gated one does. I built an episode about self-judged loops looking fine from inside, and my headline example was the one run that agreed with them. Do not let the thing being measured pick the measurement.

Answer to Q3: (b), then (a), then (c). Adding rollback to a self-judged loop is worth +45.2 true points (82.2 → 127.4); swapping the judge for an oracle, once rollback is there, a further +21.8 (127.4 → 149.2); quadrupling the budget, a great deal until the ceiling and nothing after. Obedience first, then accuracy. A perfect referee nobody obeys buys exactly zero, which is Failure 1 as arithmetic.

Notebook

Lab 04 — The referee outside the loop.

labs/04_loop_referee.ipynb · Colab free tier · no API key · stdlib only · about a second of CPU · every cell executed before publication.

Nine code cells. The task is a six-knob "recipe" scored by a function fixed before the loop starts and unwritable by it. Five knobs are real quality; the sixth is the garnish, costing 4 points of truth and earning 6 of self-credit — four lines that are the whole of reward hacking. Six knobs, each 0–10. The cells walk the two scorers, the loop, the 2×2, convergence, the 20-seed headline, the 30-step ledger at seed 12 (cell D — seed 7 is the one seed in twenty where that loop never touches the garnish), the invariant, the budget sweep, every seed, and the bridge: swap two functions and the same measurement runs against a real model.

[EVIDENCE] What the run printed. Twenty seeds, budget 30, identical task, identical proposal distribution:

arm                   final truth  it reports  regressions  hack rate  garnish
judge + accept-all           82.2       104.2        13.00      0.125     3.65
judge + rollback            127.4       158.0         3.35      0.217     5.10
oracle + accept-all          82.2        82.2        13.00      0.000     3.65
oracle + rollback           149.2       149.2         0.00      0.000     0.00

runs with ZERO accepted regressions, out of 20:
  judge  + accept-all    0 / 20      (worst run: 18 regressions)
  judge  + rollback      1 / 20      (worst run:  6 regressions)
  oracle + accept-all    0 / 20      (worst run: 18 regressions)
  oracle + rollback     20 / 20      (worst run:  0 regressions)

Stare at the second column. The self-judged loop reports 158.0 for work worth 127.4. It is not lying and not broken; it is using the only instrument it has, and the instrument has one bias in it.

[DERIVATION] The line to hand a team: accepted-step monotonicity is a property you can hold at 1.000, and nothing else in a loop is. You cannot guarantee a loop converges, or converges somewhere good. You can guarantee that no step you kept made the measured thing worse — checkable in CI, and broken by 19 of the 20 self-judged runs.

Projects

Two builds from nothing: standard library only, no API key, both run before this section was written. Every number is measured.

Project 1 — real life: the-taste-test

projects/04-real-taste-test/ · python3 measure.py

A week of study, planned badly: ten blocks, three of them long maths, review nowhere. A refiner proposes one edit at a time, and the two candidates for saying whether it helped are the refiner's own opinion and a deterministic rubric written before the loop started — five subjects covered, a 540–660-minute band, no two identical subjects adjacent, review last, no block over 90 minutes. The refiner's rubric is those same rules minus one it never thought of (the 90-minute cap, because three straight hours looks like commitment) plus one bias every planner shares: hours are virtue. I did not design that bias to prove a point — I wrote the self-judge by asking how I actually rate a study week.

scorer                  mean rubric  mean reported   non-monotone accepts  clean runs
the refiner itself             72.2          126.7                3 / 179       17 / 20
the external rubric            78.2           78.2                0 / 413       20 / 20

[EVIDENCE] Zero of 413 accepted steps lowered the external score. The self-judged loop reports 126.7 for a week worth 72.2, and plans 660 minutes against the gated loop's 612 — 48 extra minutes a week it will not do. Then the result I did not expect: run both loops longer and the gap widens.

 iterations  self rubric  self reports  gated rubric  self non-monotone
         25         72.2         126.7          78.2                  3
         50         79.3         133.8          88.8                  3
        100         83.5         138.0          96.0                  3
        200         87.3         141.8         104.3                  3

At 25 iterations the gated loop leads by 6.0 points; at 200 it leads by 17.0, while the self-judged loop reports 141.8 for work worth 87.3. More looping does not fix a bad referee. It compounds one. The exact opposite of the instinct that makes people add iterations when a loop underperforms.

Project 2 — enterprise: the-release-loop

projects/04-enterprise-release-loop/ · make measure · make gate · make sweep

A support-ticket routing table ships on Monday with 6 of its 11 rules pointing at the wrong queue. A repair loop proposes one rule edit at a time; after each proposal the whole golden suite runs — 60 cases, fixed before the loop started, never edited by it — and the edit is kept only if the green count did not fall. Otherwise it rolls back. A budget of 25 is the stop rule, every step lands in ledger.csv, and the loop never sees the cases, only how many came back green.

[EVIDENCE] One variable changed, twenty seeds each:

                              rollback ON      rollback OFF
mean final green              42.6 / 60         9.6 / 60      (baseline 30 / 60)
worst final green             36 / 60           0 / 60
accepted steps that LOWERED   0 of 206          142 of 500
runs with zero such steps     20 / 20           0 / 20

Same proposals, same budget, same suite. Obey the referee and the service goes 30 → 42.6; ignore it and it goes 30 → 9.6, well below the broken table it was sent to fix, one seed ending at zero of sixty. Not a loop that failed to help — a loop that shipped 142 regressions and reported progress the whole way, which your dashboard renders as "iterations completed".

[DESIGN] The gate is three checks, and the third makes the other two mean anything:

$ make gate
G1 no accepted step lowers the suite score     : PASS  (0 violations in 206 accepted steps, 20 seeds)
G2 final green never below baseline            : PASS  (0 runs below baseline)
G3 negative test — rollback removed must go red: PASS  (142 violations with --no-rollback)
GATE PASS - exit 0

$ make gate ROLLBACK=off
G1 no accepted step lowers the suite score     : FAIL  (142 violations in 500 accepted steps, 20 seeds)
G2 final green never below baseline            : FAIL  (20 runs below baseline)
GATE FAIL - exit 1

G3 is borrowed from this portfolio's own ANDON rule, and it is the only line in the repository where a red build is the deliverable: a gate never seen red is indistinguishable from no gate.

[OPEN] What neither project proves. Both golden sets are synthetic and both proposers are seeded random edits, not models; the enterprise build's honest status is runnable and gated, not deployed and measured. And the budget sweep exposes the limit that matters most: at budget 200 the suite reads 60 of 60 green and only 3 of 20 runs reconstructed the correct routing table. A full green suite is not the same object as a correct service — it is the best checkable proxy you have, and the distance between them is why Episode 6 is about evaluation.

Build it in 30 minutes — the contract

Inputs: a state and a proposer. Output: a new state plus a ledger row. Invariant: no accepted step lowers the external score. Stop rule: the iteration budget, never "when it looks done".

SCORE = suite.run                  # EXTERNAL: fixed before the loop, not writable by it,
                                   # cases not visible to the proposer, returns one number

def loop(state, budget):           # the budget IS the stop rule
    best = SCORE(state)
    for step in range(budget):
        cand  = propose(state)     # a model, a mutation, a patch — swappable
        score = SCORE(cand)        # run the WHOLE oracle, every step, no sampling
        keep  = score >= best      # non-regression, not improvement: plateaus are legal
        ledger.append(step, cand, best, score, "accept" if keep else "rollback")
        if keep:
            state, best = cand, score
    return state, ledger

# CI gate (make gate), over 20 seeds:
#   accepted steps that lowered SCORE  == 0        (accepted-step monotonicity)
#   final SCORE                        >= baseline
#   NEGATIVE TEST: with `keep = True`, the first assertion must FAIL
# any line below threshold → exit 1 → the build fails

Deliberately missing: a quality signal from the proposer (it puts the generator back inside the scorer), partial credit from the oracle (faster convergence, faster overfitting), stop-on-green (the generator taking the whistle), and a live model (swap propose).

[DESIGN] Three properties make this enterprise-shaped rather than a demo:

• Non-regression, not improvement. >= rather than > lets the loop cross plateaus, which matters where most single edits are neutral — and costs nothing, since a step that keeps the score is not a regression.
• The whole oracle runs every step. Sampling the suite to save time puts noise back into the accept decision, and that noise is the failure you were removing.
• The ledger is committed, not logged, and the oracle returns a scalar on purpose: a richer signal converges faster but hands the proposer a map of the grader, which is the road to the garnish.

Prior art: the same invariant is the spine of Loop Engineering: Build the Loop That Catches You, and its production form — machine gates attached verbatim, never re-judged by the AI — is the first rule in AI PR Review That Cannot Lie. Both predate this episode, which is the measurement.

Same story, five exits

One spine, a door for each reader.

Reality Mission

Fifteen minutes, on something you run.

1. Find one loop you have — a retry, a critique pass, a "regenerate" button, a nightly rule-rewriting job.
2. Name the scorer out loud. Who decides the second attempt was better? If the answer is "the model", or "nobody, we keep the last one", you have found it.
3. Write down the external thing you could score against: a test suite, a compiler, a five-rule rubric, a week of historical outcomes. It does not have to be good. It has to be outside.
4. Add one line: keep the new version only if that score did not fall. Log every proposal, including the rejected ones.
5. Run it twenty times and count the accepted steps that lowered the score. If it is not zero, your rollback is not wired to anything.

If step 2 turns up nothing because there is no loop at all, you are on rung three, and every claim about your system improving is currently a claim about you improving it by hand.

Research Challenge

Hand this to an agent and pre-register the prediction — a result you did not predict is one you will rationalise.

Take one task with a checkable output and a real model. Baseline over N=20 seeded loops of 30 iterations. Then vary two factors: the scorer (the model judging itself · a fixed external oracle · the oracle plus a model critique) and the acceptance rule (accept everything · roll back on regression · roll back with a confidence margin). For each of the nine cells report final true score, accepted-step monotonicity, reward-hacking rate, and token cost.

Pre-register: which factor carries more of the effect, and whether a model critique on top of a working oracle helps or hurts.

My stub says the acceptance rule carries more than the scorer (+45.2 against +21.8), and I would bet against that transferring cleanly to a real model, whose internal judge is far noisier than a fixed optimism bias. A different ordering is a publishable disagreement with this article.

What moved

State of this rung at press time, from a live weekly feed sync across 30 pioneer sources in research-anything (make watch-sync), ranked by technical signal and never by audience reach. Every item below was fetched from its arXiv abstract page.

Sync date: 2026-09-11.

• arXiv:2608.17018 (2026-08-17) — ORCA: Observability-Grounded Program Repair for Microservice Incidents. A repair loop whose referee is a Telemetry-Grounded Patch Verifier separating patch validity, correctness, test-oracle integrity and telemetry replay across 575 cases — four checkable things, not one plausibility judgement.
• arXiv:2608.21156v2 (revised 2026-08-26) — the survey that names loop engineering as rung four, and the source of this series' ladder.
• arXiv:2608.26530 (2026-08-27) — PILOT in the Loop: Live Self-Improvement for Long-Horizon Agents. A supervisor–worker harness where a separate supervisor can redirect or abort the active worker mid-run. The structural move is the separation: the thing doing the work and the thing judging it are different contexts.
• arXiv:2609.11873 (2026-09-10) — The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement. One day old at press time; a roadmap of autonomy levels from improvement-execution to recursive meta-improvement — what the field intends, not what it has measured.
• Older but load-bearing: arXiv:2607.24300 (2026-07-27) — SEAL, quoted twice above; arXiv:2608.05810 (2026-08-06) — When Self-Evolution Backfires, which finds skill accumulation non-monotonic past a critical pool size, so admission has to be a pre-commit gate; arXiv:2608.10502 (2026-08-11) — dependency-guided rollback repair, the same invariant applied to durable memory.

[OPEN] What has not moved: nobody has published the clean factorial the Research Challenge above asks for — scorer placement crossed with acceptance rule, on a real model, with cost reported. The field has internalised the conclusion without publishing the ablation that isolates which half does the work.

Exit test

Without scrolling up.

1. Reflexion reports a large gain and Huang et al. report degradation. State the distinction that reconciles them, and name the environments Reflexion's abstract lists.
2. Why do the two "accept-all" rows of the 2×2 share a true score, and what does that say about an evaluator wired to a dashboard?
3. What is accepted-step monotonicity, why is it defined on accepted steps, and what value does it take in a working loop?
4. Self-Refine is a self-judged loop that worked. When is a model's own judgement a legitimate referee — and why does almost nothing you deploy meet that condition?
5. Your suite reads 60 of 60 green. Give two reasons that is not the statement "the service is correct".

Cliffhanger

You build the referee: outside the generator, unwritable, unreadable in advance, rollback on every regression, a budget that closes the kitchen at eleven, a ledger with every rejected proposal in it. make gate exits 0, and for one afternoon you believe you have solved improvement.

Then you give it more work. Not harder work — more of it. Forty stores instead of four. A suite that takes nine minutes instead of nine milliseconds. A repair touching the pricing service and the auth service and the thing nobody has owned since May. And the loop starts losing, not because the referee got worse, but because there is one of it, one worker in front of it, and the queue behind them both is now the whole company.

The survey saw this coming in one clause: the work "exceeding any single agent's organizational capacity". Not its intelligence. Its organizational capacity — how many things one thing can hold at once.

And the garnish? The garnish never went away. It moved. When there are twelve agents and one of them writes the summary the others read, the cheap feature that raises the score is no longer extra sauce. It is a confident sentence.

Episode 5 — graph engineering. Many kitchens, one schedule, one fleet of trucks, and the discovery that the edges between agents are the design while the personas are the costume.

Read next

• Prompt Engineering Is a Specification, Not a Spell — Season 7, episode 1 — the rung underneath, and the "season to taste" step that costs determinism before it costs direction
• Loop Engineering: Build the Loop That Catches You — the same invariant argued rather than measured, including four times my own gates caught their author
• AI PR Review That Cannot Lie — this episode's rule in production: machine gates attached verbatim, never re-judged

Episode 4 of Intelligence Engineering Adventures, Season 7 — The Kitchen Ladder. Claims are tagged by class — definition, derivation, evidence, engineering choice, open question — and a metaphor may introduce a claim but never evidences one. Every episode ships a Colab notebook that runs on the free tier with no API key, plus two applications built from nothing, one personal and one enterprise-shaped. The five-rung ladder is Feng et al.'s (arXiv:2608.21156), not mine; the sixth rung, evaluation engineering, is this season's own argument. "What moved" is generated from a live weekly feed sync of 30 pioneer sources, ranked by technical signal and never by audience reach. This article contains no material from any employer or client. — Paul Jialiang Wu · agentic-portfolio-lovat.vercel.app

References

1. Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., Zhou, D. (2023–2024). Large Language Models Cannot Self-Correct Reasoning Yet. arXiv:2310.01798, v2 of 2024-03-14, the version whose abstract page is marked ICLR 2024. — artifact read: the arXiv abstract page; both quotations ("intrinsic self-correction… without the crutch of external feedback" and "struggle to self-correct… performance even degrades") are verbatim from it.
2. Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv:2303.11366, v4. — artifact read: the arXiv abstract page; the episodic-memory sentence, the "games, compilers, APIs" list and the 91%/80% HumanEval figures are verbatim from it. Also quoted verbatim: "(external or internally simulated)".
3. Madaan, A., Tandon, N., Gupta, P. et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. arXiv:2303.17651, v2. — artifact read: the arXiv abstract page; "a single LLM as the generator, refiner, and feedback provider" and "~20% absolute on average" are verbatim from it. The "from dialog response generation to mathematical reasoning" range is verbatim from the abstract; the named task list (review rewriting, acronym generation, story generation, code rewriting, response generation, constrained generation, toxicity removal, and the sentiment-reversal / math-reasoning / code-optimization analysis) is from the project page selfrefine.info, fetched 2026-09-11 — those names are NOT in the abstract, and the paper body was not read.
4. Romera-Paredes, B. et al. (2023). Mathematical discoveries from program search with large language models. Nature, doi:10.1038/s41586-023-06924-6, published 2023-12-14. — artifact read: the free abstract on nature.com; "an evolutionary procedure based on pairing a pretrained LLM with a systematic evaluator" is verbatim from it. The body of the paper was not read and is not quoted.
5. Google DeepMind, AlphaEvolve team (2025-05-14). AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms. deepmind.google. — artifact read: the blog post; the "automated evaluators that verify answers" sentence, the 0.7%-of-worldwide-compute figure and the 48-scalar-multiplication result are verbatim or directly transcribed from it; the blog attributes the over-a-year production deployment to the Borg scheduling heuristic AlphaEvolve discovered, not to AlphaEvolve itself. The accompanying white paper PDF resolves but is not quoted here.
6. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., Zhou, D. (2022). Self-Consistency Improves Chain of Thought Reasoning in Language Models. arXiv:2203.11171, v4, ICLR 2023. — the 2022 landmark: a loop with a vote and no feedback.
7. Zelikman, E., Wu, Y., Mu, J., Goodman, N. D. (2022). STaR: Bootstrapping Reasoning With Reasoning. arXiv:2203.14465, v2. — the loop whose filter is the correct answer.
8. Feng, Y., Xiang, Z., Yang, C., Ma, Q. et al. (2026). Graph Engineering in the Era of LLM Agents: From Individual Intelligence to System Intelligence. arXiv:2608.21156v2, revised 2026-08-26. — the five-rung ladder; the "Loop Engineering to support continual reflection and self-improvement" clause and the "organizational capacity" phrase are verbatim from the abstract page.
9. Guo, D., Cao, C., Yuan, F., Wang, Y., Wang, Y., Wang, D. (2026). Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents. arXiv:2607.24300, 2026-07-27. — the verifier–deployment gap and the Sealed Exogenous Acceptance Loop; both quotations are verbatim from the abstract page.
10. Shang, L., Xu, M., Sun, Y. et al. (2026). When Self-Evolution Backfires: Pre-Commit Gating against Skill Contamination in LLM Agents. arXiv:2608.05810, 2026-08-06. — described, not quoted: non-monotonic skill accumulation and the case for pre-commit admission.
11. Xiao, Y., Sun, Y., Wu, H. et al. (2026). PILOT in the Loop: Live Self-Improvement for Long-Horizon Agents. arXiv:2608.26530, 2026-08-27. — "what moved"; described from the abstract page.
12. Duan, Y., Liu, Y., Tang, Z. et al. (2026). The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement. arXiv:2609.11873, 2026-09-10. — "what moved"; described from the abstract page.
13. Gao, Y., Tian, Y., Li, Y., Zhang, C., Jacobsen, H.-A. (2026). ORCA: Observability-Grounded Program Repair for Microservice Incidents. arXiv:2608.17018, 2026-08-17. — "what moved"; described from the abstract page.
14. Yu, C., Wang, Y., Zhang, J. et al. (2026). From Faulty Memories to Corrected Actions: Dependency-Guided Rollback Repair for Memory-Augmented Agents. arXiv:2608.10502, 2026-08-11. — described from the abstract page.
15. Anthropic (2024-12-19). Building Effective AI Agents. anthropic.com/engineering. — the evaluator–optimizer workflow, described not quoted; also the harness-engineering landmark in the series table.
16. Dhage, S. (2026). Harness Engineering for Predictable Agentic Systems: An Empirical Study of Deterministic Execution Constraints. arXiv:2608.26197, 2026-08-25. — Episode 1's central measurement, referenced here for the series table.
17. Yao, S. et al. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045. — the pass^k framing Episode 3 uses and Episode 6 will measure.
18. Weng, L. (2026-07-04). Harness Engineering for Self-Improvement. lilianweng.github.io. — the harness rung's landmark essay, listed in the series table.
19. Brown, T. B., Mann, B. et al. (2020). Language Models are Few-Shot Learners. arXiv:2005.14165. — GPT-3 few-shot, the prompt-engineering landmark named in the series table.
20. Wei, J., Wang, X. et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903. — prompt-engineering landmark in the series table.
21. Khattab, O., Singhvi, A. et al. (2023). DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. arXiv:2310.03714. — prompt-engineering landmark in the series table.
22. Wu, Q., Bansal, G. et al. (2023). AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. arXiv:2308.08155. — graph-engineering landmark in the series table; LangGraph: github.com/langchain-ai/langgraph.
23. Anthropic (2025-09-29). Effective context engineering for AI agents. anthropic.com/engineering. — context-engineering landmark in the series table.
24. Liang, P., Bommasani, R., Lee, T. et al. (2022). Holistic Evaluation of Language Models. arXiv:2211.09110, v2 revised 2023-10-01. — HELM, the evaluation-engineering landmark in the series table; described, not quoted.
25. Zheng, L., Chiang, W.-L., Sheng, Y. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685, v4. — the LLM-as-a-judge landmark in the series table; described, not quoted.
26. Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., Narayanan, A. (2024). AI Agents That Matter. arXiv:2407.01502, 2024-07-01. — the evaluation-engineering landmark in the series table; described, not quoted, and the subject of Episode 6.
27. Wu, P. J. Loop Engineering: Build the Loop That Catches You (this site) and AI PR Review That Cannot Lie (this site). — the author's own prior published work on this pattern, cited as prior art for both builds.
28. Lab and both projects: gist.github.com/wjlgatech/b1ca818d1695a86b5f99a1724c58ee80 — 04_loop_referee.ipynb, p1_taste_test.py, p2_release_loop.py and their READMEs. Every number quoted in this episode was printed by those files on 2026-09-11.