AI-Native Series · Agent Engineering

I Scored 100% on My Own Checklist. That Was the Bug.

By Paul Jialiang Wu · agentic-portfolio-lovat.vercel.app · 2026-08-21

A lecture listed sixteen things a serious AI agent loop must have. I turned the list into a machine-checkable rubric, pointed it at my own engine, and it passed everything. A perfect score is not evidence. It is a receipt for the questions you chose to ask.

The lecture

A reader forwarded a webinar summary titled Agent Loop Engineering. It is genuinely good. It walks from prompt engineering to context engineering to loop engineering, and argues that the interesting problem has moved: not "what do I say to the model" but "how does the system act, observe, judge, and try again."

Its sixteen sections read like a specification. A loop needs a goal, state outside the model, a policy, a bounded action space, observation, and evaluation. It needs explicit exits. It needs to classify errors before retrying them. It needs a human at high-risk moments. It needs to count tokens, time, iterations and success rate.

The lecture is also unusually honest about its own status [1]:

"Agent Loop Engineering目前更适合作为一种工程实践和设计思想来理解，而不是一个已经高度标准化、具有唯一严格定义的技术标准。"

Agent Loop Engineering is better understood, for now, as an engineering practice and a design philosophy rather than a highly standardised technical standard with a single strict definition.

I appreciated that sentence, and it is also the hole. Sixteen sections of what a system must have, and not one line of evidence that any particular system has it. A taxonomy is a set of claims with the verification removed.

I have an engine that claims to be exactly this thing — a loop that generates, judges, refactors and converges. So the obvious move: stop reading the checklist and run it.

Turning an essay into a gate

My own playbook for this is one line: a rubric is a closed loop in disguise — the rubric is the score. Three rules follow from it, and they are the whole design.

The claims become data, not code. Thirteen of the sixteen sections yield checkable claims — the framing, the coding-agent walkthrough and the closing produce none — and those turn into a YAML file, one entry per claim. The checker reads that file and nothing else, so the published score can never drift away from the claims it says it is scoring.

Evidence is observed, never asserted. Each claim carries a probe: a symbol that must exist at a real file and line, or a test that is actually executed — not merely present on disk. The output records where it looked and what it found.

"I could not look" is not "it works." If a probe cannot run, the item is unmeasured: excluded from the percentage and blocking. This is the rule that most checklists quietly break, because an unrunnable check is indistinguishable from a passing one if you let it be.

- id: L16-safety-terminal
  claim: "A safety failure is terminal and unbypassable, whatever the score."
  probe: {type: grep, pattern: "BLOCKED_SAFETY", paths: ["src/loopeng/loop/convergence.py"]}

# ... two entries omitted; each real item also carries the lecture section it came from

- id: L19-hitl-unbypassable
  claim: "An unattended run cannot pre-confirm its own result."
  probe: {type: pytest, node: "tests/test_run_contract.py::test_contract_can_never_disable_the_human_gate"}

I ran it [4]. Twenty-three claims, twenty-three passes, gate green.

The moment to be suspicious of yourself

A perfect score felt good for about four seconds.

Then: I wrote these questions. I wrote them after spending an hour inside the codebase. At no point did I write a question I thought would come back red. What I had built was not an assessment; it was a well-engineered mirror, with test coverage.

This has a name in the measurement literature. Donald Campbell's 1979 study of planned social change [2] argues that the more a quantitative indicator is used to make decisions, the more pressure it comes under to distort the very process it was meant to monitor. (The famous one-sentence version of that argument circulates everywhere; the original is paywalled, so I am describing his argument rather than putting words in his mouth.) Steven Kerr's On the folly of rewarding A, while hoping for B [3] makes the operational half of the point: systems reliably produce the thing you actually score, not the thing you meant.

Both are usually invoked to explain other people gaming a metric. The sharper version is closer to home. When the author of the rubric is also the owner of the system, no gaming is required. Nobody cheats. The selection happens earlier and more comfortably, at the moment you decide which questions are worth writing down.

Going back for the questions I skipped

So I reread the lecture looking only for claims I had not encoded, and I chose them by a deliberately hostile criterion: include it if I expect it to fail.

Five came back — the four I expected to fail, and one I did not.

ReAct-style interleaving (section 5): reasoning and acting alternating at fine granularity. My loop does not do this. Human takeover (section 12): the lecture says a person may take a running task over, not merely approve the result. Mine cannot. Manager-worker hierarchy (section 11): a manager agent decomposing work for workers. Mine has no such thing. Task success rate (section 14): the lecture asks for tokens, time, iterations and success rate. I had the first three per run, and the fourth did not exist anywhere.

The fifth was the one I would have bet against myself on: does the engine surface failures that recur across runs, so it stops repeating a mistake? It does — recurring_failures, shipped months earlier. It passed. Twenty-three claims became twenty-eight, and I got exactly one piece of good news I had not selected for.

The score moved from 23/23 to 25/28 — 24/28 the instant the claims went in, 25 once I closed the cheapest miss in the next section — and the gate went red. That failure is the first output of this exercise I actually believe.

The one I fixed, and the two lines that decide whether a number is honest

Of the four misses, one was cheap and genuinely useful: a success rate across runs. The engine recorded every finished run's outcome; nothing ever divided.

Writing it took ten minutes. Deciding what counts took longer, because a success metric is the easiest thing in a system to flatter.

A run still in flight is excluded from the denominator. Counting it as a failure punishes you for having work in progress; counting it as a success is a lie with a delay on it. It is reported separately and left out.

A run halted by the safety gate counts as a failure. This one is uncomfortable and it is the important one. A safety block means the work did not ship. If the rate quietly forgave those, the metric would reward exactly the outcome the safety gate exists to produce — and I would have built a number that goes up when my engine is stopped for being dangerous.

And when nothing has finished, it returns nothing at all rather than zero. Zero reads as "we tried and failed." Nothing reads as "we have not measured." Those are different sentences and a dashboard should never confuse them.

The headline line of the audit file, generated rather than typed — docs/rubrics/AUDIT.md:

**Conformance: 25/28 (89%)** verifiable claims implemented · 3 declared gaps · 0 unmeasured · gate **FAIL**

The rubric holds thirty-one items in total: twenty-eight verifiable claims, scored 25/28, plus three declared gaps scored separately and never folded into that fraction.

Misses with reasons are still misses

The other three stayed red, and there was a real temptation to reclassify them. Two of them are, honestly, deliberate divergences rather than oversights.

ReAct puts the acting model in charge of judging its own next step. My engine's load-bearing rule is that the thing which builds never grades — so adopting ReAct wholesale would break the one property I trust most. Manager-worker coordination is likewise a choice: I use an explicit dependency graph with cycle rejection, which can be inspected before anything runs, instead of a manager improvising at runtime.

Both are defensible. Neither is a pass. So the rubric now carries a rationale field that records the reasoning beside a red mark, and there is deliberately no mechanism to convert a rationale into a green one. A checklist where the author can explain failures away is a checklist that scores nothing — which is precisely the failure I had already committed once that morning.

The third, human takeover, gets no clever framing. It is a real gap. My gate is end-of-run approval; there is no channel into a loop already in flight. Worth building. Not built.

Gaps that fail when they close

There is one more category, and it is the piece I would steal for any checklist you own.

Three things the lecture implies and my engine genuinely lacks — a typed failure taxonomy, branching search, and a span-level trace schema — are written into the rubric as declared gaps. Their probes assert the absence. They pass while the gap is real, and they fail the moment the gap silently closes.

That sounds backwards until you have watched a documented limitation quietly become false. Someone adds tracing in a side branch; the "we don't have tracing" line in the README stays for a year; a reader plans around a constraint that stopped existing. Making the gap list executable means the document cannot rot without something going red.

Patterns and anti-patterns

Patterns. Write the questions you expect to fail, first. Keep the claims as data and let exactly one program read them. Make evidence a file and a line, or a test that actually ran. Treat an unrunnable check as blocking, never as passing. Let a rationale sit next to a red mark without turning it green. Make your declared gaps executable so they break when they stop being true.

Anti-patterns. Authoring a rubric for a system you own without a single question you fear. Reading a taxonomy as though it were an audit. Counting in-flight work in a success rate. A metric that improves when your safety gate fires. Reporting zero for "not measured." A gap list that only a human remembers to update. And the broad one behind all of them: treating a green check as evidence without asking what it was able to see.

The mechanism, in one sentence

The mechanism is selection at authoring time: a test suite's ceiling is set not by how rigorously it runs but by which questions its author was willing to write down, and that choice happens before any code executes, in private, with no red mark to make it visible.

The first principle: a checklist that cannot fail its author is not an assessment, it is a mirror — so the only trustworthy score is one produced by a list you did not get to curate after seeing the answers.

What this article does not claim

• 89% is not a quality score. It is conformance to one webinar's taxonomy, which is neither exhaustive nor authoritative — the lecture says so itself.
• The rubric is still mine. Adding questions I expected to fail improves it; it does not make it independent. A rubric written by someone else, or derived mechanically from the source text, would be stronger, and I have not done that.
• Three red items are unresolved, and one of them (human takeover) has no plan yet beyond being named.
• One engine, one author. The audit runs against my own repository; your failure modes will differ.

The takeaway

Every serious team is about to write one of these. The AI-agent field is producing checklists at a remarkable rate — maturity models, readiness frameworks, sixteen-section taxonomies — and almost all of them will be scored by the people who built the thing being scored.

You will get a good number. You will deserve very little of it.

The cheapest possible fix takes an afternoon: before you run your checklist, write down the three questions you are hoping nobody asks. Put them in. If the result is still green, you have learned something real. If it goes red — as mine did, that same morning, on my own engine — then for the first time the number is worth quoting.

References

1. AI开发进入新时代：Agent Loop Engineering — 讲座总结, DataApplab / AI聘 (info@aipin.io), webinar summary received 2026-08-20. Quoted verbatim from the source email; the English rendering beneath it is mine and is labelled as a translation, not as the author's words.
2. Donald T. Campbell, Assessing the Impact of Planned Social Change, Evaluation and Program Planning, 2(1), 1979, pp. 67–90. Described, not quoted: the original is paywalled, and the single sentence everyone attributes to it is reproduced so widely second-hand that I would not put it in quotation marks without reading the paper.
3. Steven Kerr, On the folly of rewarding A, while hoping for B. Originally Academy of Management Journal, 18(4), 1975; the copy I verified is the reprint in The Academy of Management Executive, February 1995, 9(1), p. 7. Title and publication details checked against that PDF; I cite the reprint because it is the version I actually read.
4. The rubric, the checker and the full audit output: docs/rubrics/agent-loop-engineering.yml, scripts/audit_loop_rubric.py, and docs/rubrics/AUDIT.md in github.com/wjlgatech/loop-engineering-anything. Every number in the second run — 25/28, three declared gaps, zero unmeasured, gate FAIL — is reproducible by running that script. The first run is reproducible too, against the archived rubric: --rubric docs/rubrics/archive/agent-loop-engineering.v1.yml returns 23/23, gate PASS. That archive is a reconstruction committed after the fact, not a recovered original, and it says so in its own header.

Written 2026-08-21. The 23/23 run and the 25/28 run are the same engine on the same day; only the rubric changed. One correction, because this article would not survive its own standard otherwise: when I first wrote this, the 23-item rubric had never been committed. The only trace of 23/23 anywhere was prose I had written — a commit message, a YAML comment. An independent reviewer caught it, and they were right: my headline number was asserted, not observable, in a piece arguing that the difference is everything. It is now committed as a labelled reconstruction and re-runnable in one command [4].