AI-Native Series · Agent Engineering · Loop Engineering in Practice

AI PR Review That Cannot Lie

Evidence per claim, "unmeasured" over silent, and a path envelope for auto-merge. A build report from a repo where human review had become the bottleneck, written for the co-founder who asked for it and for anyone about to hand a merge button to a model.

By Paul Jialiang Wu · agentic-portfolio-lovat.vercel.app · 2026-09-08

What this report covers, and why an AI PR review needs a different design

This is a report on one week of work. A team I co-build with named "PR review by a human" as the single bottleneck to iteration speed, and assigned me two same-day items: deploy an automatic reviewer on the repository, and write down the review standard it enforces. The repository is a quality-management system (QMS) for factories, where every record may later be pulled as evidence in a customer audit. That raises the stakes for review in a specific way: a merged bug is not a bug, it is a false record.

The design question is not "can a model review code". Models review code well enough to be useful. The question is what an AI PR review must be forbidden from doing so that a team keeps trusting it after the tenth PR. My answer is four prohibitions and one envelope, and the whole article is those five things with their evidence:

1. The reviewer may not make a claim without a file:line citation. Unbacked claims are dropped, and the number dropped is printed on the brief.
2. The reviewer may not be silent. No key, a model refusal, an API error: each produces a posted comment that says "unmeasured" and "this is not 'no problem'".
3. The reviewer may not re-judge what scripts already decided. Tenant isolation, secrets, migrations, foreign keys, bilingual docs, tests, and the golden case library are attached to the brief verbatim from the scripts that computed them.
4. The reviewer may not leave the human-only list empty without saying why. "Nothing for a human to judge" must come with its reason, or the absence reads as "the reviewer did not run".
5. The reviewer may only trigger a merge inside a path envelope it did not choose, and only as one of five AND-ed conditions.

Readers: AI engineers who run one maintainer and many agents; founders who need to decide what "auto-merge" may touch; and students who want to see how a gate is made honest rather than merely installed. The analogy comes after the problem, and it is one picture from one domain: a restaurant kitchen with three passes before a plate leaves.

The bottleneck, measured before anything was built

Before proposing a reviewer I pulled the repository's own numbers, because "review is slow" and "review does not exist" need different fixes. Twenty-four merged PRs. Twenty-four authored by the same person. Twenty-three merged by their author. Zero review records on the eight PRs I sampled. Nine open PRs from other contributors, all with zero reviews, the oldest waiting sixteen days. The median merge time of six minutes in the history was not fast review; it was an author merging themself.

Two more facts shaped the design. The CI had silently failed for eleven days the previous month, every job dying in two seconds with zero steps because the account's Actions minutes had run out, and nobody had noticed because red-with-no-steps looks the same as red-with-a-failure from a distance. And the repository was a free private plan, which meant no branch protection at all: any green or red PR could be merged by anyone, and the only thing preventing it was memory.

Google's study of its own practice analysed logs for nine million reviewed changes [5]; a practice that has run zero times produces no such logs, and no such learning. So the actual situation was a review process that had never run once, about to receive nine external PRs through a single person, on a repository with no enforcement layer. A reviewer that occasionally hallucinated a finding, or occasionally failed quietly, would land on a team with no habit of checking. That is why the prohibitions come before the prompt.

The mental model: three passes in a restaurant kitchen

One picture carries the design, and it is the kitchen on the cooking ladder, at the "running a restaurant" rung. A plate leaves a serious kitchen through three passes, and the passes are different in kind, not in strictness.

The reason a kitchen works this way is the reason the reviewer works this way. The moment the sous-chef's note becomes the thing that sends plates, cooks start cooking for the note. The moment an AI verdict becomes the merge condition, contributors start writing PRs that satisfy the AI. Keeping the note as a note, and the scale as the scale, is what keeps both honest.

The mechanism: four things the reviewer is not allowed to do

1. No claim without a file and a line

The model returns structured output constrained by a JSON schema [4]: a list of findings, each with a rule id, a severity, a claim, and an evidence array. Before anything is rendered, a filter walks every finding and keeps only evidence strings matching path:line where the path is either in the PR's changed files or exists in the repository at that commit. A finding with no surviving evidence is removed, and a counter increments. The brief ends with the line "N claims without file:line evidence were dropped".

Two details matter. The path may point at a file the PR did not touch, because "this conflicts with existing code" is exactly the kind of finding whose evidence lives in an old file; restricting evidence to the diff would delete the most valuable category. And the count is printed, not hidden, because a reviewer that quietly discards half its own output is telling you something about its calibration that you want to see.

The system prompt says this in one sentence: claims without evidence will be discarded by the system, so say less rather than invent. The filter is what makes that sentence true rather than hopeful.

2. Unmeasured is a posted state, never silence

The first version of the brief, when there was no API key, printed the machine-gate results and nothing else. It looked like a reviewer with no objections. That is the most dangerous output a reviewer can produce, because from the outside "I found nothing" and "I did not run" are the same comment.

The fix is a branch that cannot be skipped. If there is no credential, if the model returns a refusal stop reason, if the API raises, or if the structured output fails to parse, the brief renders a section headed "unmeasured" with the exact reason, followed by the sentence "this is not 'no problem'", followed by the machine gates and a note that every human-only item now needs a human. The workflow still posts it. A unit test pins each of those phrases, so a refactor cannot quietly restore the silent version.

3. Machine gates are attached, not re-judged

Seven gates exist, G1 to G7: an AST pass over routers for tenant filtering and permission dependencies; a secret-shape scanner over added lines that reports positions and never echoes values; model-to-migration pairing; a foreign-key gate over every *_id column; bilingual documentation pairing; the test suites; and a golden case library where deterministic modules must reproduce their standard outputs. The reviewer itself runs the four scripts that decide six of them, because the submission-gate script covers secrets, migration pairing and bilingual docs in one pass; the seventh, the test suites, runs in its own CI jobs and the reviewer does not repeat it. So a brief shows four script lines under "machine gates", and those four lines cover G1 to G5 and G7. The reviewer records pass or fail with the last six lines of output and hands the result to the model with the instruction: these are decided, do not re-decide them.

The reason is not that the model would get them wrong. It is that a second opinion on a deterministic check is noise, and noise on a brief is what makes people stop reading briefs. The most cited field study of modern code review, at Microsoft, found that reviewers spend their effort on understanding rather than on defect-hunting, and put it this way in the abstract:

"Our study reveals that while finding defects remains the main motivation for review, reviews are less about defects than expected and instead provide additional benefits such as knowledge transfer, increased team awareness, and creation of alternative solutions to problems."
— Bacchelli and Bird, ICSE 2013, author's copy [6]

If human reviewers are mostly building understanding, then an AI brief that repeats a scale reading is spending the reader's attention on the one thing a script already settled. The model's attention goes where scripts cannot reach: whether a closing action writes an audit record, whether a create endpoint can reach a terminal state, whether a router is doing business logic, whether the PR description's claims have pasted evidence.

4. An empty human-only list must say why

The brief's first section is "needs your judgment", and it is the only section a code owner has to read. When it is empty, the model must fill a companion field with the reason, and the renderer prints "empty; reason: …". Without that, an empty section is indistinguishable from a model that never considered the question. This is the same principle as prohibition two applied at a finer grain: absence is never allowed to carry meaning.

Rules as data, and three principles that became rules

The 26 rules live in one JSON file, not in the prompt and not in the documentation. Each rule has an id, a tier, a title, what to check, which gate or person decides it, and the source it was derived from. The documentation page's table is generated from the file by a --render-rules command, and a --check-doc command in CI fails if the table in the document differs by a character from what the file renders. The prompt receives the same rendered table. Three consumers, one source, no drift.

Seven rules are machine tier and each names the script that decides it; a test asserts those scripts exist, because this repository had earlier documented a secret scanner that had never been written. Six are human tier, lifted from the project's own 15-item module checklist: the items that ask "what real factory problem does this solve" cannot be answered from a diff. Thirteen are AI tier. Ten of them are the checkable residue of the project's engineering laws: audit records on critical actions, terminal states unreachable from create endpoints, five UI states, router-only-orchestrates, human-signs-every-AI-reachable-write, single concern, implementation log present, tests for changed pure logic, gate coverage, and description-matches-evidence.

The last three were added because my co-founder asked for review "according to our OOP, loop-engineering and graph-engineering principles", and a principle that lives only in a conversation is not a rule. So:

R12 is the article's own rule applied to itself, which is why the next two sections exist.

The envelope: paths decide, not confidence

The team's request was that PRs which meet the bar merge without a person in the way. The earlier design said the AI must never be a merge condition. Both are right about something, and the resolution is not a compromise between them; it is a different variable. The question "should the AI's verdict cause a merge" has a bad answer in both directions. The question "where is the blast radius low enough that a wrong merge is a remake rather than a refund" has a data answer.

The envelope is two lists of path prefixes in the rules file. Allowed: documentation, plans, the golden case files, test directories, and anything ending in .md. Denied, and deny wins over allow: the workflow directory, the scripts directory, the application package, the migrations, frontend source, the Makefile, and the two agent-guide files. A Markdown file under the workflow directory is denied. A test file under the application package is denied. The model does not see this list as a choice; it is applied after the model has answered.

Tier-1 then requires five things at once: every changed path inside the envelope; every gate script green (the four lines on the brief; G6, the test suites, is a required status check that GitHub's auto-merge itself waits on); the AI review measured with a verdict of merge; zero findings at block or warn severity (a third level, info, is shown on the brief but never affects the tier); and an empty human-only list. When all five hold, the workflow approves the PR as the Actions identity and enables GitHub's auto-merge, so the merge fires when the required checks finish. When any fails, the PR gets a "needs human" label and the brief lists every unmet condition, not just the first. That last point came from a unit test: the first implementation returned early on the first failure, which means a contributor would fix "path outside envelope", re-run, and only then learn about the warn-level finding. Three reasons should print as three reasons.

Two GitHub settings make the bot's approval count: the repository must allow Actions to approve pull requests [3], and branch protection [1] must not require the approval to come from a code owner, because the Actions identity is not one. Auto-merge itself is GitHub's feature [2]; the workflow only enables it. The code-owners file still routes review requests to people. The protection script defaults to one required approval with the code-owner requirement off, and prints both repository settings back after applying, because a free plan silently ignores the auto-merge setting and a script that does not read back is a script that lies.

Day one: the reviewer's first two verdicts were about itself

The workflow shipped in a PR, and that PR was the first thing it reviewed. There was no API key in the repository yet. The brief said so: "unmeasured: ANTHROPIC_API_KEY missing (repository secret not configured). This is not 'no problem'." Below it, the four gate scripts, all green, and the tier line: needs human, because the changed paths included the workflow and the scripts directory, and because the AI review was unmeasured.

My co-founder added the secret. The next PR, the one adding the envelope, was the second thing it reviewed. The brief said: "unmeasured: AuthenticationError 401, invalid x-api-key." Same tier line, same two reasons, plus a third: the key the platform had been given was not a key the platform accepted.

I am recording this as the day-one result rather than as an embarrassment because it is the design under its intended load. A reviewer that had stayed quiet on both runs would have produced two PRs that looked reviewed and were not. Instead, two PRs carried an explicit statement that the AI had not run, why, and what that meant for the human. The mechanism did not merge either of them on its own; both were outside the envelope, and the second condition failed anyway. Zero false greens is the number; the scope is two runs, one repository, one day.

There was a second silent failure on the same day, one layer down. Branch protection is a paid feature on private repositories, and the upgrade had been purchased. The API answered a paid upgrade with a 403 for ten straight minutes of polling, and the auto-merge setting patched back as false. The protection script exits non-zero with the upgrade URL when that happens; it does not pretend. The account-level cause is still being diagnosed as this is written, which is the honest status of the last condition, and it is on the page rather than in a footnote for the same reason the brief posts "unmeasured".

Patterns, anti-patterns, and the first principle

Patterns that held. Evidence as a schema field, filtered by existence, with the drop count printed. Failure states rendered as sections, never as absence. Deterministic checks attached from their scripts, not re-derived by the model. Rules in one data file rendered into the prompt and the documentation by the same function, with a drift check in CI. Blast radius expressed as path prefixes with deny-wins, applied after the model has answered. Every unmet condition listed, not just the first.

Anti-patterns this replaced. A reviewer that approves. A brief that omits a section to mean "no issue". A confidence score standing in for a citation. A rule that lives in a prompt and a second copy in a document. A silent skip when the credential is missing. An early return that reports one reason when three are true. Each of these looked like a reasonable first draft and each would have made the reviewer easier to ignore by the tenth PR.

The first principle, in one sentence. A review signal is only worth reading if its absence and its failure are both visible, and its claims can be checked without trusting it; everything else in this article is that sentence applied to a merge button.

Build it in 30 minutes

This is the contract, not the code. The code is repository-specific; the contract transfers. Thirty minutes is the budget the contract sets for a reader who copies the table into a fresh repository; it is not a time I measured. The shell here was written in one sitting, and the tests, the two documents and the day-one runs took the rest of the day.

# The whole loop, from a PR event
diff, omitted   = collect_diff(merge_base, changed_files)      # risk-ordered, capped, omissions listed
gates           = run_machine_gates(base)                       # scripts decide; last 6 lines kept
review          = call_model(meta, files, diff, gates, rules)   # structured output, one call, no retry
review, dropped = filter_findings(review, files)                # path:line or it is gone; count kept
tier, reasons   = classify_tier(brief, envelope)                # five conditions, all reasons listed
brief           = render_brief(...)                             # never silent; unmeasured is a section
post_comment(pr, brief); apply_tier(pr, tier)                   # upsert; approve + auto-merge only on tier-1

Same story, five exits

What is measured, and what is not yet

• Measured. Fourteen unit tests pin the evidence filter, the unmeasured rendering, the envelope's deny-wins semantics, the five-condition tier logic and its cumulative reasons, and the doc-drift check. Two live runs on real PRs produced two "unmeasured" briefs and zero silent outputs. Both PRs were classified needs-human for listed reasons.
• Not yet measured. The negative test (a planted defect producing a block-level finding) is open as a PR and will run once a valid key is present; until it goes red, the reviewer is installed but not live by its own rule. The Tier-1 positive path (a docs-only PR approved and merged by the bot) is blocked on branch protection, which is blocked on an account-level plan issue. No numbers exist yet for precision of AI findings on real code; the evidence filter is a floor, not a measurement.
• Illustrative. The kitchen. It maps one-to-one and it is not data.

Read next

• AI Review Told Me to Make It Worse — the earlier episode on why a reviewer's advice needs an evidence bar, from the other side of the table
• The Agent-Ready Codebase — the gates this reviewer attaches verbatim were built there first
• Prompt Engineering Is a Specification, Not a Spell — the same series, and the same argument one rung lower: constrain the free-text step

Part of the AI-Native Series · Agent Engineering. Numbers in this article are scoped where they appear; the kitchen is a model, not a measurement. The rules file, the reviewer, the workflow, the protection script and the tests described here exist in a private repository; the shapes are reproduced in full above so that nothing depends on access to it.

References

1. GitHub Docs. About protected branches. docs.github.com — required status checks, required reviews, and the plan availability of protection on private repositories.
2. GitHub Docs. Automatically merging a pull request. docs.github.com — the auto-merge behaviour Tier-1 relies on.
3. GitHub Docs. Managing GitHub Actions settings for a repository. docs.github.com — the setting that lets a workflow approve a pull request.
4. Anthropic. Structured outputs. docs.anthropic.com — the schema-constrained response the evidence filter operates on.
5. Sadowski, C., Söderberg, E., Church, L., Sipko, M., Bacchelli, A. (2018). Modern Code Review: A Case Study at Google. ICSE-SEIP, DOI 10.1145/3183519.3183525. research.google · author's copy (PDF) — described, not quoted: 12 interviews, a 44-respondent survey, and review logs for 9 million reviewed changes.
6. Bacchelli, A., Bird, C. (2013). Expectations, Outcomes, and Challenges of Modern Code Review. ICSE 2013. author's copy (PDF) — the quotation above is from the abstract of this copy, checked word for word against the PDF; the version of record at IEEE may differ in typography.