AI-Native Series · Agent Engineering · Loop Engineering in Practice

The AI Review Loop Report: Eleven Silent Days, One Chat Card, and a 24% Ceiling

What broke, what we built, the six decisions and why, what the reviewer caught in its own repairs, and the next moves, each with the observation that would prove it wrong.

By Paul Jialiang Wu · agentic-portfolio-lovat.vercel.app · 2026-09-24

What this AI review loop report covers, and who it is for

This report continues a line of work in one repository. AI PR Review That Cannot Lie built the reviewer: every claim cites a file and a line, and "unmeasured" is a posted state instead of silence. The ANDON series turned that into five rules, and its fourth episode, the auto-merge envelope, ended with an honest sentence: the door that lets a bot merge had never opened. In parallel, the Visionboard series built the instrument I run my projects on, and its last episode, the scoreboard, closed with three bets for the next quarter, each with a dated falsifier. The second bet was a sensor that cannot be silent.

This report is what happened when those designs met a real week. On 2026-09-23 my co-founder, who owns the quality domain, and I agreed on three things for the AI review loop: the reviewer judges two dimensions (does the change do what the requirements say, and does it follow the engineering rules); it escalates to a human only when it is unsure, which we guessed would be 5 to 10% of pull requests; and every notification goes to the team chat instead of e-mail. The same day I measured the guess. It was 100%.

The report has three parts. Report: the problem, the solution, and the six decisions with the reasoning behind each. Reflect: what the build taught, including what the reviewer caught when it reviewed its own repairs, and one lesson about delegation that cost me an afternoon of trust. Project: the next moves, small and big, short and long, each tied to a Visionboard bet or an ANDON door and each with the observation that would prove it wrong.

Readers: engineers who run an AI reviewer or are about to; founders deciding what "auto-merge" is allowed to touch; students who want one complete loop, from failure to measurement to design to what is still unproven. The codebase is private; every mechanism is reproduced here in enough detail to rebuild it.

One picture: the pass, the sous-chef, and a bell in the chef's pocket

One picture will carry the report, and it continues the kitchen from the first report, on the cooking ladder's "running a restaurant" rung. Every plate leaves through the pass. A sous-chef tastes it and clips a note to the ticket. The head chef reads the note, tastes one spoonful, and either sends the plate or sends it back. Cold starters the sous-chef is sure about can go straight out.

For eleven days the sous-chef was off shift. The kitchen's rule said "never pretend you tasted it", so on every plate the sous-chef's stand-in clipped a note: not tasted. The rule was obeyed perfectly. The notes sat at the pass. The chef was in the dining room, phone in hand, and never walked past the pass. Every plate waited for the chef, which is safe, and the kitchen's promise that most plates would flow without him quietly became zero plates. Nobody lied. Nobody heard.

The fix is a bell in the chef's pocket that rings in two different tones.

Where the picture breaks: a real sous-chef who is off shift is missed at once, because the kitchen slows. A reviewer that is "off shift" costs nothing visible, because the human path still works. That is the whole problem, and the reason the grey ring had to exist.

Part I · Report — the problem, measured before it was fixed

Later that same day, US time (the meeting ran at 22:04 in China), I pulled every pull request the reviewer had labelled, and read the reason printed in each summary. Not the label: the label for "the AI is unsure" and the label for "the AI never ran" were the same label.

Share of pull requests that reached a human: 47 of 47. Scope: one repository, the reviewer's labels from its first labelled pull request to 2026-09-23. [MEASURED]

Read again a day later, with four more pull requests: 51 labelled, 0 auto-merged, 4 escalated by a reviewer that ran, 47 where it never ran. The shape did not change, only the count.

Five whys, each answered from the repository or its logs:

1. Why did every pull request reach a human? Because 45 of 47 summaries said the reviewer had not run.
2. Why had it not run? Every call since about 2026-09-12 returned the same error: the organization's monthly spend limit was reached, and access would return on 2026-10-01.
3. Why was the limit reached? The reviewer shared one API key, and one $100 monthly limit, with everything else I run on that machine, including a batch job that translates articles into four languages. The billing console shows spend per key, not per job, so I cannot say which job spent most of it; I can say the reviewer was the one that starved.
4. Why did nobody notice for eleven days? The review job is set to not block merges when it fails, so the workflow showed green. The "unmeasured" summary was posted on each pull request, exactly as the never-silent rule requires, and acted on by no one in eleven days, because nobody opens a summary on a pull request they are about to review by hand anyway.
5. Root cause: the reviewer's own failure had no path to a human ear. "Never silent" was satisfied on the page and violated in the world.

The repository's lessons file already had an entry from two weeks earlier saying almost exactly this, unmeasured needs someone watching it, with a next step that never landed. A lesson recorded as prose is a lesson the prose can stop being true about. It is the thesis of ANDON episode 3, rules as data rather than prose, and I apparently needed to learn it a second time.

Part I · Report — the solution: an AI review loop that reports to the channel people live in

The team had already decided that the chat app (Feishu) is the only inbox. So the loop has four hops: GitHub runs the reviewer, a script turns its summary into a chat card, a button on the card returns the human to GitHub, and the merge event sends one more card back.

The card kinds, and what decides each:

The first end-to-end run, 2026-09-24: once the organization limit was raised, the reviewer ran on a real pull request, raised three warnings and two questions for a human, and the team chat received an orange card with its questions; after the fixes, the merge produced a blue card. Every hop was seen working on a live pull request, not only in a test.

Part I · Report — six design decisions, and the reasoning behind each

A seventh decision is a refusal: the loop does not decide anything. It carries decisions. The reviewer suggests, the rules file decides what may auto-merge, and a human decides everything else. That is the design of the Visionboard applied to a team: agents propose and report, and the human stays above the loop. In the small-team episode, the human-only field became a role with separation of duties. Here the role is "code owner", and the card is how the role gets asked.

Part I · Report — the 24% ceiling: what auto-merge can actually reach

A loop that routes decisions faster does not tell you how many decisions a human must make. So the second measurement asked a different question: of the last 60 merged pull requests, how many touched only files the auto-merge rules allow, such as docs, plans, tests and golden test cases? That is the ceiling. No reviewer, however good, can auto-merge a change the rules exclude. One of the 60 had exactly 100 files, the most GitHub returns per pull request, so its list may be cut short; it is counted as unmeasured, which leaves 59.

Path-eligible for auto-merge: 14 of the 59 measurable pull requests among the last 60 merged (24%). Adding frontend styles, components and views: 15. Adding the tooling folder: 20. Scope: one repository, measured 2026-09-24; a ceiling, not a forecast, because eligible changes still need green gates and a clean review. [MEASURED]

The biggest blocker was a surprise, and it is ours. Nineteen of them touched AGENTS.md, the handbook the coding agents read before they work. The repository has a rule that every feature change must update that handbook, so humans and agents never read stale instructions. And the auto-merge rules correctly forbid a bot from merging changes to the handbook, because changing it changes every agent's behaviour. Put the two rules together and every feature change needs a human, by design. The robots must update their handbook, and the robots may not approve updates to their handbook. Both rules are right. The target was wrong.

Widening the frontend would have added one pull request, because real frontend changes usually touch protected files too. Widening the tooling folder as well (the what-ifs are cumulative) adds five more, and that folder is where the gates and the reviewer themselves live: letting a bot merge a change to the grader is letting the cook grade their own plate. So the proposal for the next team decision is to keep the rules as they are, replace the target, and run a two-week pilot. It has one more gate that has nothing to do with software: GitHub refuses branch protection on this repository, answering "Upgrade to GitHub Pro", so the bot can approve but cannot merge. Until that changes, the auto-merge rate is zero by construction, which is the same door the envelope episode found closed.

Correction, 2026-09-25. This report first said the door needed a $4 monthly plan. The owner already had that plan. Every other private repository on the same account accepted branch protection; this one is a private fork of an older repository, and a fork inherits these features from its upstream network, not from its owner. The error message named a fix that had already been applied. The real fix is detaching the fork. It is the same lesson as the rest of this report, one level down: I had read the state from the message instead of measuring it, and the measurement took a handful of API calls.

Part II · Reflect — what the reviewer caught when it reviewed its own repairs

The fixes to the loop were themselves pull requests, so the reviewer, freshly revived, spent its first afternoon grading the code that revived it. It ran ten times on four of my pull requests. It asked for more work five times and approved five, and every round, approvals included, listed something real. Here are the catches in order, because the order is the lesson.

My favourite catch was not the reviewer's, and I found it only because a new test failed. When the reviewer script crashes, the alarm card explains why in words I wrote: "the script itself failed; this is not a budget or 401 problem". The card's explainer searches the text for "401" to recognise an invalid key. It found my disclaimer and told the reader to rotate the key. The alarm had diagnosed itself from the sentence that said what it wasn't. The fix was to reword the disclaimer; the lesson was that I had written a keyword detector and then fed it keywords.

Catch 6 has a second half I am less proud of. The reviewer posted it as "hold" on the proposal's last update, the chat received an orange card, and I merged anyway, because the tests were green and I did not open the card. The loop worked end to end; the human at the end of it did not read. The fix was a follow-up pull request, the ceiling moved from 23% to 24% once the truncated change was set aside, and the repository's implementation log now carries the rule I broke: before merging, read both the test result and the reviewer's latest verdict, and write down why when you overrule a hold.

Catch 5 is the one I would have missed and a colleague might not have mentioned. The pull request said "here is the grep output, the line exists". The grep output had been shortened by me, and the reviewer, reading the diff context, counted one more match than I showed. It then marked the claim "unverified" instead of arguing with it. That is the first report's rule, no claim without evidence, enforced on its own author.

Part II · Reflect — lessons learned

1. Never silent is necessary, not sufficient. A state must reach an ear. The reviewer obeyed the rule for 45 pull requests and helped no one. The test for any alarm is not "was it posted" but "who read it within a day". This is the Visionboard's second bet, "a sensor that cannot be silent", with a stronger clause: a sensor also cannot be unheard.
2. Measure the denominator before you argue about the ratio. We spent part of a meeting designing for 5–10% escalation. The real number was 100%, and 45 of those were not escalations at all but a broken sensor. Put the "did not measure" cases in their own bucket before computing any rate; when the bucket is everything, the rate is "unmeasured", not 0% or 100%.
3. A shared budget lets the least important job starve the most important one. The reviewer shared a budget with a batch translation job and every other project on the machine, and it was the one that starved; the console could not say which job spent the month, because the whole spend sat on one old key they all used. Give the critical path its own budget, and track spend per job, not per key.
4. The gate you write for others will grade you. Ten review rounds on my own repairs, five requests for more work, and the one I ignored was right. The day the reviewer came back, the author it reviewed most was the one who built it. That is the best evidence so far that it is worth roughly $0.25 a review (an estimate: the key's cost column read $1.96 after about eight reviews).
5. Deliver results, not navigation. The agent that did this build for me sent me through a billing console four times: "Settings, then Limits", "Rate limits", "switch workspace, then Billing". The last destination does not exist. The vendor's own help centre says "The default Workspace is not editable and cannot be removed." [5] I was the executor of a plan the planner had not checked. The rule it wrote for itself afterwards: verify the destination before asking a human to go there, hand over one link and one action, and if a step cannot be verified and is not essential, drop it and deliver the result without the human. The people lesson is the same shape: when you delegate upward, the person above you should receive a plated dish, not a recipe and a map to the pantry.

Patterns, anti-patterns, and the mechanism

• Pattern: two tones for two faults. "The judge is unsure" and "the judge is absent" go to different cards, colours and people.
• Pattern: the router is dumb on purpose. Deterministic classification from text the system wrote, never from the model's opinion of itself.
• Pattern: loud by default. Silence is an explicit, listed decision per case; anything unlisted rings.
• Anti-pattern: the green failure. A job allowed to fail without blocking shows green while doing nothing. Allowing it not to block merges is right; letting that also make it invisible is the bug.
• Anti-pattern: a target without a ceiling. A ratio goal set before measuring what the rules allow.
• Mechanism: jidoka. Toyota describes it as the ability, when an abnormality occurs, for "the machine or equipment can detect the abnormality and stop automatically, or the operator can stop the line by pulling the stop cord themselves" [1]. In the kitchen, the cord is the sous-chef's note and the board is the bell: pulling the cord only helps if it rings where the chef is standing. The chat card is that bell.
• First principle, one sentence: a check is only as real as the shortest path from its failure to a person who can act on it.

Part III · Project — the next moves, small and big, short and long

Each move is tied to a Visionboard bet or an ANDON door, and each carries the observation that would prove it wrong, dated, in the style of the scoreboard episode. The Visionboard's falsifier date for its second bet, the silent sensor, is 2026-10-07; the first two moves report against it.

What I am deliberately not doing: approving from chat, widening the auto-merge rules before the pilot, or building a second reviewer to review the first. Each is interesting. None closes a loop that is open today.

Build it in 30 minutes: the review-loop contract

The contract, not the code. Thirty minutes is the budget this contract sets for a reader copying it into a repository that already has an AI reviewer posting a summary; it is not a time I measured.

Same story, five exits

Reality mission

Pick one automated check your team trusts. Break its input on purpose tonight, a bad key or an empty file, and see where the failure shows up tomorrow morning, and who notices first. If the answer is "on a page nobody opens" or "nobody", you have the problem this report is about, and the grey card is a 30-minute fix.

Read next

• AI PR Review That Cannot Lie — the reviewer this loop carries, and the kitchen this report continues
• Jidoka for Repositories: the Never-Silent PR Gate — the rule this report finds necessary and not sufficient
• The Auto-Merge Envelope: Blast Radius Is a Path List — the door that is still closed, now with a ceiling and a price
• The Visionboard Scoreboard — the three bets, including "a sensor that cannot be silent", that Part III reports against
• Life GPS for a Small Team — the human-only field as a role; here, the code owner the card asks

Part of the AI-Native Series · Agent Engineering. Numbers are scoped where they appear and come from one private repository's own measurement commands on 2026-09-23 and 2026-09-24; the per-review cost is my estimate from the billing console's per-key cost column, not a metered average. The kitchen is a model, not a measurement.

References

1. Toyota Motor Corporation. Toyota Production System. global.toyota — the jidoka passage quoted above, checked word for word against the page on 2026-09-24.
2. GitHub Docs. Expressions: status check functions, always. docs.github.com — the quoted definition, checked against the page source on 2026-09-24.
3. GitHub Docs. Control the concurrency of workflows and jobs. docs.github.com — cancel-in-progress, the reason a superseded review is cancelled and why that case is quiet.
4. GitHub Docs. About protected branches. docs.github.com — branch protection rules; on this repository, a private fork, the API answered "Upgrade to GitHub Pro" even though the owner's account was already Pro.
5. Anthropic Help Center. Creating and managing Workspaces in the Claude Console. support.claude.com — the quoted note on the default workspace, checked against the page on 2026-09-24.
6. Anthropic. Workspaces (Claude Platform documentation). platform.claude.com — described, not quoted: every organization has a default workspace; spend limits are set per additional workspace, under the organization's own limit.
7. Feishu Open Platform. Custom bot usage guide. open.feishu.cn — the webhook the cards are posted to; described, not quoted.