AI-Native Series · Agent Engineering

An AI Reviewed My Repo and Told Me to Make It Worse

By Paul Jialiang Wu · agentic-portfolio-lovat.vercel.app · 2026-08-12

It graded me 8.6 out of 10 and proposed twelve upgrades. Half of them were real. One would have quietly undone the best decision in the codebase — and the only way I could tell the difference was to stop reading the review and go read the code.

The review

Someone ran a frontier model over loop-engineering-anything — my engine for driving a generated tool from Grade F to Grade A against an independent judge — and sent me the output. It was good. It was better than most human code review I have received: structured, specific, complimentary in the places I had actually worked hard. It proposed twelve upgrades, then a further backlog of twenty-three items.

It also had a headline: your architecture is too CLI-shaped. Its closing recommendation was to introduce a formal artifact type — a fixed list of the kinds of thing the engine can converge — so that the repo's promise of "anything" would become technically true:

"Keep CLI as the first proven lane, but abstract the engine around 'artifact convergence,' not 'CLI convergence.'"

Here is the thing. Every claim the review made about my code cited exactly one source: my README.

That is not a gotcha. It is the most useful sentence in this article. A reviewer who can only read the front of the house will review the front of the house, and if the front of the house says "this makes CLIs," the reviewer will correctly conclude that this makes CLIs. The finding was real. It just wasn't a finding about my architecture. It was a finding about my documentation, wearing an architecture's clothes.

So I did the boring thing and spent an hour reading my own source before touching a line of it.

What the code said back

The proposal was to add something like this to a central schema file:

artifact_type: Literal["cli", "mcp_server", "agent_skill", "workflow", "spec"]

A closed list. Every new kind of thing the engine can improve becomes an edit to that one file, reviewed by whoever owns that one file.

But the engine already had a generalisation seam, added eight weeks earlier, and it worked the opposite way. A domain is a small plugin that answers three questions — is this target mine, how do I build the first version, and who referees it — and it announces itself to a registry at start-up. Adding a new kind of target is a registration. Nothing central changes. The module docstring I wrote at the time reads: "a new domain arrives as a registration, never as an edit to router.py or the controller…"

The reviewer's suggestion would have taken an open system and closed it. Not maliciously, and not stupidly — it simply could not see the seam, because the seam is not in the README.

Six of the twelve proposals were real gaps. Five were things I had half-built and under-documented. One was this. I wrote up the adjudication item by item and committed it next to the code [5], because a review you argue with in private is a review you will re-litigate in six months.

The real gap, and the trap inside it

The biggest genuine hole was embarrassing once named. Everything you tell my engine to do — what to improve, how good is good enough, how many attempts, how much money, when to stop — existed only as command-line flags. Typed into a terminal. Gone when the window closed.

So there was no artifact for the ask. I could show you a beautiful proof of what a run achieved and not show you what it was asked for. Fixing that is obvious: put the ask in a file, commit the file.

And that is where the trap is, because the obvious fix is also the most common way software lies.

A missing letter, a thousand servers

Kubernetes is the system that runs a large fraction of the world's server fleets. You tell it what you want in a YAML file. In April 2023, two Google engineers announcing a validation feature on the Kubernetes blog described what used to happen when you got that file slightly wrong [1]:

"Before Kubernetes v1.8 (!), typos, mis-indentations or minor errors in YAMLs could have catastrophic consequences (e.g. a typo like forgetting the trailing s in replica: 1000 could cause an outage, because the value would be ignored and missing, forcing a reset of replicas back to 1)."

Read that again slowly. You asked for a thousand. You wrote replica instead of replicas. The system did not argue. It accepted your file, discarded the word it did not recognise, found no instruction about how many servers to run, and helpfully fell back to one.

Your file was accepted. Your fleet was not.

The exclamation mark in that quote is the engineers' own. They put it there because "before v1.8" is a long time to have shipped a system that treats a misspelling as a silent vote for the default.

This is not a Kubernetes flaw; it is the default behaviour of nearly every config parser ever written, and it comes from a genuinely well-intentioned engineering tradition — be liberal in what you accept. In 2023 the Internet Architecture Board published a document walking that advice back, on the grounds that tolerance "is no longer considered best practice in all scenarios." Martin Thomson and David Schinazi write [2]:

"Time and experience show that negative consequences to interoperability accumulate over time if implementations silently accept faulty input."

Their reasoning is the part worth stealing. Tolerance is not free and it is not local: once one implementation quietly accepts a malformed thing, everything built afterwards is built against the quiet acceptance rather than the specification. They call it a chain reaction — "Overapplication of the robustness principle therefore encourages a chain reaction that can create interoperability problems over time" — and note it hurts most in a system's early days, when today's quirk becomes tomorrow's compatibility requirement.

A young governance file is exactly a young protocol.

The permission slip

Here is the model I used, and you can run it on any config file you own.

A school field trip needs a permission slip. Three things make it a real control rather than a piece of paper:

1. It can only ask for things the school actually does. You may request the bus. Writing "and a helicopter" on the slip does not summon a helicopter.
2. If you write something the office does not recognise, they hand it back. They do not put you on the bus while letting you believe a helicopter is coming. The worst outcome is not a rejected slip; it is an accepted slip and a child on the tarmac.
3. You cannot sign it yourself. Your parent may add "no peanuts." Neither of you may cross out "requires a signature."

Those three sentences are the entire design of the contract I shipped. In the code they read like this.

Rule 1 — it compiles, it does not extend

Every key that configures anything turns into a setting the loop already obeys. The file cannot introduce a knob; it can only fill in one that exists. Eight keys at the top level, ten inside the budget block, one each inside the gate and evidence blocks, and nothing else. The flag-driven path and the file-driven path then hand the engine the identical object: one code path, two doors.

Rule 2 — an unknown word is an error, not a shrug

This is the Kubernetes lesson, enforced:

$ loop-anything contract check loop.yaml
Error: loop.yaml.budget: unknown key(s) ['max_iteratons']; accepted:
['compression_interval', 'max_iterations', 'max_tool_retries', 'max_wall_seconds',
'min_score_gain', 'plateau_patience', 'plateau_pivots', 'target_grade',
'target_score', 'token_budget']. A contract may only declare what the engine enforces.

The parse fails. The run does not start. And the error message is a to-do list — it names every key that is accepted, so a typo takes seconds rather than a debugging session. (That output is one line in a terminal; it is wrapped here to fit the page.)

The second half of that rule is the one that actually took judgement. The reviewer's proposed schema contained a block like this:

safety:
  forbidden:
    - delete_production_data
    - expose_credentials

That reads wonderfully. It would sail through any compliance review. And in my engine, nothing whatsoever would read it — the real enforcement lives elsewhere, in a sandbox that refuses shell metacharacters and jails writes to a workspace. Accepting that block would have created a second, decorative safety system whose only function is to be quoted in a slide.

A rule nobody enforces is a No Smoking sign in a building with no smoke detector. It does not stop the fire. It just means the fire is now non-compliant.

So the parser rejects it, by name, with the reason. The same goes for a domain: key (the run path cannot yet force a domain, so accepting it would do nothing), and for per-dimension scoring weights (the judge owns its own rubric; a file written by the thing being graded must never steer the grading). Each becomes legal the moment something enforces it. The error message is the roadmap.

Rule 3 — the gate only tightens

My engine has a rule that predates all of this: when a run reports success, that is a claim, not a shipped fact, until a human confirms it. There is a flag that supplies a human's yes on an attended run. There is deliberately no flag that disables the gate — and an unattended, scheduled run that tries to pre-approve its own work is refused outright.

The reviewer's contract design included a human-gate block with a boolean: require_confirm_before_ship: true. Which means it can also be written false. And the contract is authored by the same person who wants the run to ship.

So in mine, that field may be omitted or set true. Setting it false is a parse error with an explanation. A contract may add reviewers; it may never remove the signature. The permission slip cannot un-require the parent.

Worth being precise about, since this article is about precision: that field compiles to nothing at runtime. The gate is already on by default, and no code path reads the key. Its entire job is to make false unwritable — the one place in the contract where the parser is not a doorway to a setting but the enforcement itself. By my own Rule 2 that is a key with no reader, and I keep it only because the thing it refuses is worth refusing loudly. (The gate has exactly one bypass, and it is not a caller's to use: an environment variable the build platform owns, never honoured on a scheduled run.)

Evidence: the part that makes it not a promise

The last section of the file lists what the run must prove — the before-and-after grades, the per-dimension diff, the regression tests recorded when a fix was accepted. Seven such items exist, and every name you write must be one of the seven, because those are the only things a real run can produce.

Then a command checks the file against the actual recorded run:

$ loop-anything contract evidence loop.yaml --run 12
  present  grade_trajectory
  MISSING  regression_tests
Error: run #12 is missing declared evidence: regression_tests

Exit code 1. My proof packs omit rather than fabricate a field they have no source for, so a missing item is a genuine gap between what was promised and what was proved — in that example, a run that converged without ever recording a test that would catch the bug coming back.

That is the difference between a governance artifact and a governance aesthetic. One of them can print a failure with your name on it.

Why the reviewer was wrong in a way that matters

There is a second reason I did not simply execute the backlog, and it is not about this particular review.

The engine's central discipline is that the thing which builds is never the thing which grades. Arjun Panickssery, Samuel Bowman and Shi Feng gave that discipline a measurement in 2024 [3]:

"One such bias is self-preference, where an LLM evaluator scores its own outputs higher than others' while human annotators consider them of equal quality."

They go further, and this is the sharp part: fine-tuning models to be better at recognising their own text produced "a linear correlation between self-recognition capability and the strength of self-preference bias." The better a model gets at knowing its own voice, the more it favours it. Self-evaluation does not fail randomly; it fails directionally.

An AI review of your repo is not the same thing as an AI grading its own output — the reviewer had no stake in my code. But the underlying move is identical, and it is the move this whole engine exists to refuse: accepting a verdict from a party whose evidence you have not checked. The review's evidence was my README. Once I checked it against the source, five of its eight dimension scores moved on my own re-scoring — three down, two up. The dimension it scored lowest, generality, is the one I raised most; the dimensions I lowered were safety, observability and enterprise readiness, which it had been kinder about than the code deserves.

Patterns and anti-patterns

Patterns. Make the parser fail closed, and make the failure message enumerate the legal options. Let a policy file only tighten a default, never loosen one. Name the thing that will read each field before you accept the field. Check declared evidence against produced evidence with an exit code. Ship the adjudication of a review next to the code, so the argument is version-controlled.

Anti-patterns. A config key with no reader. A safety block in a file that no safety code opens. A boolean that lets the author waive their own review. A schema whose closed list of types must be edited to extend the system. Executing a backlog because it was numbered and well-formatted. And the oldest one: treating "it parsed" as "it was understood."

The mechanism, in one sentence

The mechanism has a name: the enforcement gap — the distance between what a configuration file declares and what any line of code actually reads. Every silent config bug and every decorative compliance artifact lives in that gap, and the gap is invisible by construction, because a file that is ignored looks exactly like a file that is obeyed.

The largest instance on record is not a config file at all. Do Not Track was a header your browser sent, once you switched it on, to say you did not wish to be followed. Browsers sent it faithfully for years. No server was ever obliged to read it. Its working group closed on 17 January 2019 and its specifications were retired [4] — a decade of a perfectly transmitted preference that nothing was required to obey.

The first principle, then: a declaration that cannot fail is not a control, it is a costume — so the only honest way to add governance to a system is to add the refusal at the same time.

What this article does not claim

• No benchmark. I have not measured whether the contract reduces real-world misconfiguration. It is a design argument plus one implementation, not a study.
• The review was not bad. Half of it was correct, and its best idea — measuring how often my loop reports success when a stronger judge would disagree — is the thing I have not built yet, and the thing I most need.
• One engine, one author. The three rules come from my own codebase. The Kubernetes and IAB evidence supports the principle, not my particular implementation of it.
• Test counts are not quality. The suite went from 543 to 600 passing tests. That measures coverage of the new code, not that the design is right.

The takeaway

The upgrade shipped in a day. The lesson took the whole day too, and it was not about YAML.

When a very fluent system hands you a very well-organised list of what is wrong with your work, the list's fluency tells you nothing about its grounding. Ask what it read. Then go read the thing it could not.

And when you finally write the governance file — the contract, the policy, the standard, the checklist — write the refusal first. If you cannot describe the input that would make it print an error with your name on it, you have not built a control. You have printed a sign.

References

1. Jeffrey Ying and Antoine Pelisse (Google), Kubernetes 1.27: Server Side Field Validation and OpenAPI V3 move to GA, Kubernetes Blog, 24 April 2023. kubernetes.io/blog/2023/04/24/openapi-v3-field-validation-ga. Quoted verbatim; the parenthetical exclamation mark is theirs.
2. M. Thomson and D. Schinazi, Maintaining Robust Protocols, RFC 9413, Internet Architecture Board (IAB) stream, June 2023. rfc-editor.org/rfc/rfc9413. Both quotations verified verbatim against the RFC text (Sections 2 and 4.1).
3. Arjun Panickssery, Samuel R. Bowman and Shi Feng, LLM Evaluators Recognize and Favor Their Own Generations, arXiv:2404.13076, 15 April 2024. arxiv.org/abs/2404.13076. Quotations from the abstract.
4. W3C Tracking Protection Working Group. The group page states: "The working group is currently closed. It closed on 17 January 2019," and lists the Tracking Preference Expression (DNT) documents under retired specifications. w3.org/2011/tracking-protection.
5. The item-by-item adjudication of the review, committed beside the code it argues about: docs/solutions/external-eval-2026-08-12.md. The contract itself is src/loopeng/contracts/run_contract.py, its design note is docs/solutions/run-contract.md, and the annotated example is docs/examples/loop.yaml, in github.com/wjlgatech/loop-engineering-anything.

Written 2026-08-12. The review that prompted it was produced by a frontier language model at a reader's request; I have kept its argument and not its identity, because the point is the sourcing, not the model.