AI-Native Series · ANDON · Agentic PR Review and CI/CD · Episode 2 of 5
Jidoka for Repositories: the Never-Silent PR Gate
Agentic PR review and CI/CD, episode 2. In 1896 a loom learned to stop itself when a thread broke. Most CI pipelines never learned that lesson: a gate that could not run looks exactly like a gate that found nothing, and both are green. This episode is the N in ANDON. It builds a gate with three states instead of two, a response clock that runs on GitHub's cron rather than anyone's memory, and then reports the one failure the design did not catch, because it had no voice at all.
1-minute takeaway — what you'll walk away with
What this is. Episode 2 of ANDON, a five-part series on agentic PR review and CI/CD. This one is the rule never silent, built into a real repository as a never-silent PR gate: an AI reviewer with three states (pass, fail, and unmeasured), a response-time bot that labels pull requests and rewrites one dashboard every day without anyone remembering to, and the honest report of a failure that had no state at all.
Why it matters. For an engineer: two-state gates lie by omission, because "could not run" renders as "found nothing", and the flood makes that omission routine. For a founder: the enterprise data says human review coverage fell from 89 to 68 percent while throughput doubled, so the signal you are losing is exactly the signal a silent gate hides. For a student: this is the oldest idea in quality control, a machine that stops itself, applied to the newest problem.
What you can do after reading. Give every automated judgment three states and a posted output for each. Put a response clock on GitHub's cron with a policy file and a dashboard that is overwritten, never appended. And adopt the merge rule this episode paid for: merge on the workflow run's own conclusion, never on the checks list, because a failure with no check-run is invisible there.
ANDON — the series spine
- A · Assess — The PR flood is measured, not felt episode 1
- N · Never silent — Jidoka for repositories this episode
- D · Data, not prose — The rule that cannot drift episode 3
- O · Only in the envelope — Blast radius is a path list episode 4
- N · Negative test — A gate is installed only once it has gone red episode 5
What this episode covers, and why a never-silent PR gate comes second
Episode 1 measured the flood: merged pull requests on GitHub more than tripled between January 2023 and mid-2026, and inside one enterprise that measured itself, per-capita throughput reached 2.09 times its baseline while the share of PRs with a human review fell from 89 to 68 percent [1, 2]. It ended with the machine gate: a scale with no opinions that decides only what a script can decide. This episode is about what the scale does when it cannot weigh.
That case is not rare. An AI reviewer needs a credential; credentials expire, get pasted wrong, get rotated. A gate that reads the pull-request body gets no body on a push event. A workflow file can fail to parse. A bot that writes a dashboard finds the dashboard feature switched off. In every one of those cases a two-state gate, pass or fail, produces its pass state by default, because nothing ran and nothing failed. The reader of the PR sees green. The literature says the reader is increasingly the only human who will look at all: a 2026 study of how humans review AI-generated PRs found most receive no human review, with the reviews that do occur mostly bot-authored [3], and a manual inspection drawn from 11,048 agentic PRs found that 5.5 percent showed no visible interaction trace whatsoever [4].
So the rule for this episode is short: a gate has three states, and the third one talks. The worked example is the same private quality-management repository as episode 1, where the AI reviewer's first eleven live runs all landed in the third state, a response-time bot ran for the first time, and, on the same day, two PRs were merged on a failure that produced no state at all. The five whys at the end ask why the third state must be a posted comment and not an absence, and the answer goes back to a loom in 1896.
The oldest version of the rule: a machine that stops itself
Toyota's own account of its production system starts with a loom. Sakichi Toyoda's 1896 power loom carried what the company describes as "a new weft-breakage automatic stopping device (developed in 1896)": when the weft thread broke, the loom stopped rather than weaving a flaw into the cloth [5]. The Type G automatic loom stopped for a broken warp thread as well as a broken weft, and the first one came off the production line in November 1925 [6, 7]. Taiichi Ohno's gloss on the principle, jidoka, is "automation with a human touch." [8] The point of the loom was not that it detected a broken thread. It was that detection and stopping were the same event: the machine could not continue in a broken state without a person knowing.
Two more pieces of the lineage matter here. On 16 May 1924, Walter Shewhart sent his supervisors at Western Electric a memorandum proposing the control chart [9], the tool that separates chance variation from what he called assignable causes; his 1931 book states the postulate directly: "Assignable causes of variation may be found and eliminated." [10] A cause that produces no signal cannot be found, and therefore cannot be eliminated. And Toyota's use of the andon system, the cord any line worker may pull to stop production, dates back to 1950 by the company's own account; the description is that "a team member on the production line can pull the cord to flag up the issue, stop the process and receive immediate assistance." [11] Stopping is a right and a duty, not an accusation.
The kitchen picture from episode 1 holds. The scale is pass one. What this episode adds is the mixer that halts when a spring breaks, and the cook who says "the scale is broken" out loud instead of guessing the weight. Neither is a judgment about the dish. Both are the kitchen refusing to proceed in an unknown state.
| Kitchen | Repository |
|---|---|
| The mixer that halts when a spring breaks | The reviewer's third state: "unmeasured", posted on the PR with the reason |
| The expediter's ticket timer, which buzzes on its own | The response clock: a cron on GitHub, 48 hours to a first response, 7 days to a decision |
| A ticket that never printed, so nobody is cooking it and nobody knows | A workflow that fails to parse: no check-run, nothing on the checks list |
The numbers on silence: how little of the flood gets a human look
The reason the third state matters more in 2026 than in 2016 is that the human backstop is thinning. The convergent practice of modern code review, as measured in 2013, was "performed regularly and quickly just before the code is committed", usually by two reviewers, with median completion times of roughly fifteen to twenty-one hours [12]. Google's engineering guide sets the norm in one sentence: "One business day is the maximum time it should take to respond to a code review request" [13], and its 2018 study of about nine million reviewed changes found a median full-review latency under four hours [14].
Against that baseline, the agent-era numbers read as a collapse of coverage rather than of speed. In the enterprise study, AI review coverage rose from about 19 to 84 percent of PRs while human coverage fell from 89 to 68 percent [2]. Among agentic PRs on GitHub, most get no human review at all [3]. Of 717 manually inspected PRs, a third of the rejections carried no observable rationale, and 5.5 percent showed no visible interaction trace at all [4]. The vendors' own defaults tell the same story from the other side: Anthropic's managed code review posts a neutral check rather than a blocking one and averages fifteen to twenty-five dollars a run [15]; Cursor's Bugbot reports a neutral status unless you opt into failing on unresolved issues [16]. A neutral check is a polite form of silence: it is there, and it decides nothing.
None of the studies in the dossier found an organisation that responded to agent volume with a formal review service level. The thirty-month record shows coverage falling instead. That gap is what the second half of this episode builds against.
The build: three states, and a clock on GitHub's cron
1. The AI reviewer's third state
The reviewer from the precursor article and episode 1 reads the diff, runs the machine gates, asks the model for structured findings, drops any finding without a file:line citation, and posts one comment per PR. The change that makes it never silent is small in code and large in consequence: there is no path through the program that ends without a posted comment. If the credential is missing, if the platform returns an authentication error, if the model refuses, if the structured output fails to parse, the brief renders a section headed "unmeasured" with the exact reason, followed by the sentence "this is not 'no problem'", followed by the machine-gate results and a note that every human-only question now needs a human. The workflow posts it anyway. A unit test pins each phrase so a refactor cannot quietly restore the two-state version.
The live record is the evidence, read from the PR comments on 2026-09-09 (the first two runs are described in [17]). Between 2026-09-06 and 09-09 the workflow ran on every pull request in the repository from the one that introduced it onward, eleven in all, and posted a brief on all eleven; all eleven said "unmeasured", first because the repository had no key, then because the key it was given was rejected with a 401; none stayed silent. The loop audit built at the end of this run reads those two facts straight from the GitHub API and reports them as the criteria "every PR carries a brief" and "unmeasured is posted, never silent", both passing, while the criterion "the AI has actually measured a PR" stays unmeasured until a valid credential exists. The audit is not allowed to round that up, and the third incident below is what happened the one time it did.
2. The response clock
The repository's own proposal had measured the problem on 2026-09-02: every merged PR self-authored, none reviewed, the oldest open one at 398 hours. Any fix that depends on a person remembering would reproduce that record. It sat unimplemented for a week. On 2026-09-07 the domain co-founder opened three pull requests; by the morning of 09-09 they had waited fifty-seven hours with no maintainer response, which was the breach the proposal had been written to prevent. That morning the bot was built.
It is a pure function plus a cron. The function takes a pull request, the current time and a policy file, and returns one of five states: ok, needs-first-response, overdue-decision, exempt, or draft. Maintainer activity is any comment, review, merge or close by a name in the policy's maintainer list, with three exclusions that came out of the unit tests: the author's own comments do not count even when the author is a maintainer; the AI reviewer's comments do not count, because a machine's "unmeasured" is not a person's "I have seen this"; and the dashboard's own comments do not count. Forty-eight hours without a maintainer response is the hard threshold; seven days without a decision is a soft one. Two PRs are exempt by number with a reason written in the policy: a legacy PR from before the gates existed, and the deliberately open negative-test PR from the precursor article.
The cron runs daily on GitHub's infrastructure. On each run it applies or removes two labels and rewrites one dashboard issue in place, found by title, so the board is a single overwritten surface rather than a growing thread of alerts. The first live run, dispatched by hand as the reverse validation the proposal demanded, labelled the three co-founder PRs and left the two exempt ones alone. It then exited non-zero, because the repository had issues disabled and the dashboard could not be created. That was correct behaviour: the contract has two halves, labels and board, and a run that delivered one half reported the missing half as a failure rather than a success. The repository setting was changed, the bot re-ran, and the board appeared with three breached rows and two exempt rows. The bot now names that failure in plain words when it happens again.
The state with no voice: merging on an invisible red
Here is the failure the design did not catch, reported as it happened. A pull request merged that morning added a comment to the CI workflow file. The comment contained, as prose, a literal pair of empty expression braces, meant to say "do not interpolate expressions into the shell". GitHub evaluates expressions everywhere in a workflow file, comments included, and an empty expression is a parse error. From that commit on, the workflow could not run.
What that looks like from outside is the important part. A workflow that fails to parse produces a run with zero jobs, a conclusion of failure, and a name that is the file path rather than the workflow's name. It produces no check-runs. The command that lists a pull request's checks, which is what I had been using to decide whether to merge, showed nothing for it at all. It showed the one other workflow's result and stopped. I merged that pull request, and then a second one, on a red that had no row.
The loop audit caught it the same day, because its practical criterion for CI reads the latest run's conclusion on the main branch, not the checks list, and that run said failure with the file's name on it. The fix was a change to the comment, a unit test that fails if any workflow file contains an empty expression, a lesson in the repository's own record, and a new merge rule for me: judge a pull request by its workflow run's conclusion, never by the checks list, because the state that hurts is the one the list cannot show.
The same afternoon the audit caught a second failure of the same family in the opposite direction. The new provenance gate requires an agent-authored PR to paste its verification output in the PR body. On a push event, after a merge, there is no pull-request body, so the CI step wrote an empty file and the gate read the emptiness as "the section is missing" and turned the main branch red. That is a two-state gate reading absence as failure instead of as pass, which is safer, and still wrong. The fix gave it the third state: an empty body is "no body to check", reported as unmeasured, and the CI step only writes the body on pull-request events. Both incidents are in the repository's lessons file as entries 8.27 and 8.28, and both have a test.
Then the audit did it to itself. Its first run inside CI came back with one red criterion and one green one, and the green one was the defect. The red was honest: the job had no test runner installed, the audit tried to run the tests, could not, and scored that as a failure. The green was the compounding check, the one that compares each count against a baseline commit. The CI checkout had no local main branch, the command that finds the baseline failed, and the code's fallback took the error text as if it were a commit id and read every baseline count as zero. Zero to anything is growth, so the check passed. A silent default inside the tool built to catch silent defaults. The fix is the same shape as the other two: a missing test runner is "unmeasured", a baseline that cannot be resolved is "unmeasured", and five unit tests hold it there. That is entry 8.29, and it is the reason the line above says the audit is not allowed to round up rather than that it does not.
Five whys: why "could not run" must be a posted state and not an absence
| # | Why? | Because | Evidence |
|---|---|---|---|
| 1 | Why does a gate need a third state at all? | Because in a two-state gate, "did not run" and "found nothing" produce the same output, and the reader cannot tell them apart. | Eleven of eleven briefs in the worked case were "unmeasured", read from the PR comments on 2026-09-09; under a two-state design all eleven would have read as clean. The first two runs are in [17] |
| 2 | Why is that worse now than it used to be? | Because the human who would have noticed is increasingly not looking: coverage fell 89 to 68 percent in the enterprise study, and most agentic PRs get no human review. | [2, 3] |
| 3 | Why must the third state be posted rather than logged? | Because a state that lives only in a log is a state nobody reads at the moment of the merge decision; the loom stops on the floor, not in a ledger. | The 1896 loom's stop is the detection [5]; the invisible-red incident showed a failure that existed in the run log and nowhere the merge decision looked |
| 4 | Why must it never count as pass? | Because an assignable cause that is scored as chance can never be found and eliminated; the audit that rounded "unmeasured" up to "pass" would report a healthy loop that has never actually run. | Shewhart's postulate [10]; the loop audit keeps "AI has measured a PR" unmeasured on purpose |
| 5 | Why does the response clock belong in the same episode? | Because a PR nobody has answered is the human version of silence, and Google's one-business-day rule and the andon cord both say the same thing: the right to stop the line is only real if someone pulls it every day. | [13, 11]; the first run labelled three PRs at fifty-seven hours |
The root, in one sentence: a gate's failure to run is itself a measurement, and a measurement that is not posted where the merge decision is made has not been taken.
Build it in 30 minutes: the never-silent contract
Thirty minutes is the contract's budget for a reader who copies it; it is not a time I measured. The third state took one branch in the reviewer and one test; the response bot took a morning; the three incidents took an afternoon each.
| Element | Specification |
|---|---|
| Three states | Every automated judgment returns pass, fail, or unmeasured, and every state has a posted output on the pull request. Unmeasured carries the reason verbatim (missing credential, 401, refusal, parse failure, no input for this event type) and the sentence that it is not a pass. |
| Empty lists carry reasons | A section that is empty because there is nothing to report must say why; an empty section with no reason is indistinguishable from a section that was never computed. |
| Event-aware inputs | If a gate's input differs by event type (a PR body exists on pull-request events and not on push), "input unavailable" is unmeasured, never fail and never pass. |
| Response clock | A pure classifier over (PR, now, policy) with states ok · needs-first-response · overdue-decision · exempt · draft. Policy as data: hours to first response, days to decision, maintainer list, ignored comment markers, exemptions with a written reason. A cron on the platform's own scheduler. Two labels, one overwritten dashboard found by title. |
| Two-halves rule | If the bot's contract has two outputs (labels and board), delivering one and not the other is a failure that names the missing half, not a success. |
| Merge rule | Merge on the workflow run's conclusion for the branch, never on the checks list. A run with zero jobs named after the file is a parse failure and is red. |
| Guard | A test that fails if any workflow file contains an empty expression, and one that fails if a gate script lacks a self-test (episode 5). |
| Deliberately missing | No retry on unmeasured; no escalation beyond the label and the board; no paging. Each would add a path that can itself fail silently. |
# The classifier is the whole bot; everything else is plumbing.
def classify(pr, now, policy):
if str(pr.number) in policy["exempt"]: return "exempt" # with a written reason
if pr.is_draft: return "draft"
responses = [t for who, t, body in pr.activities
if who in policy["maintainers"] and who != pr.author
and not any(m in body for m in policy["ignore_comment_markers"])]
if not responses:
return "needs-first-response" if age_h(pr, now) > policy["first_response_h"] else "ok"
return "overdue-decision" if age_h(pr, now) > policy["decision_days"] * 24 else "ok"
# The merge rule, as a command (the checks list cannot show a parse failure):
# gh run list --workflow ci.yml --branch <branch> --json conclusion → merge only on "success"
Same story, five exits
| Reader | The one decision | The one action |
|---|---|---|
| Student | "It didn't complain" is not the same as "it checked". | Find one automated check you rely on and make it print something when it cannot run. |
| Engineer | Three states, each posted. | Add an "unmeasured" branch to your reviewer and a test that pins its wording. |
| Founder | A response clock is a policy file and a cron, not a person. | Write the two numbers (hours to first response, days to decision) and the exemption list, and let the platform run it. |
| Executive | The failure you cannot see is the one that costs the most. | Ask what your merge decision is based on, and whether that surface can show a workflow that never started. |
| Investor | The loom that stops itself is worth more than the loom that weaves faster. | Ask a team to show you a gate reporting "could not run", on a real PR. |
Patterns, anti-patterns, and the first principle
Patterns that held. Three states with a posted output each. Empty lists that carry a reason. Inputs classified by event type. A classifier that is a pure function over a policy file. One overwritten dashboard. A reverse validation run before the bot is called live. The two-halves rule.
Anti-patterns this replaced. A reviewer whose missing-key path printed only the machine gates and looked like a clean review. A service level that lived in a proposal for a week while three PRs aged past it. A merge decision based on a list that cannot show the failure that matters. A gate that read a missing input as a failing input.
The mechanism, named. Every code path in the reviewer and the bot ends in a post; an absence is structurally impossible, so the only way to be silent is for the platform itself to fail before the code runs, which is exactly the case the merge rule covers.
The first principle, in one sentence. A judgment that could not be made is a fact about the system and must be recorded where the next decision is made; silence is not a state, it is the absence of one.
Reality mission
Episode 1 promised that by episode 2 the response bot would have run on the three PRs that were at fifty-seven hours. It has: they carry the label, the board lists them, and the run that created the board is recorded in the dossier with its number. By episode 3, the loop audit that caught the invisible red will have been re-run on a green main branch and its report committed, and the three co-founder PRs will either have a first response or the board will still say they do not.
Read next
Episode 2 of ANDON · Agentic PR Review and CI/CD. Every number is scoped where it appears; the kitchen is a model, not a measurement. The reviewer, the response bot, the loop audit and the three incidents described here exist in a private repository; the shapes are reproduced in the contract so nothing depends on access to it. The four-horizon dossier this episode cites from is logged in the portfolio's plans directory.
References
- GitHub, How pull request limits are cutting down the noise, 2026-06-18. github.blog
- He, Agarwal, Denisov-Blanch, Azaletskiy, Koyejo, Vasilescu, AI Writes Faster Than Humans Can Review: A Longitudinal Study of an Enterprise 2x Mandate, arXiv:2607.01904, 2026-07-02. arxiv.org
- Duma et al., These Aren't the Reviews You're Looking For, arXiv:2605.02273, 2026-05-04. arxiv.org
- Peralta et al., Why Are Agentic PRs Merged or Rejected?, arXiv:2605.22534, 2026-05-21. arxiv.org
- Toyota Motor Corporation, Toyota Production System. global.toyota
- Toyota Industries Corporation, Sakichi Toyoda. toyota-industries.com
- Toyota Motor Corporation, 75 Years of Toyota, Type G. toyota-global.com
- Autonomation, Wikipedia, quoting Ohno 1988, pp. 6, 58. en.wikipedia.org (secondary; Ohno's book was not fetched)
- Best, Neuhauser, Walter A Shewhart, 1924, and the Hawthorne factory, Quality & Safety in Health Care, 2006. pmc.ncbi.nlm.nih.gov
- Shewhart, Economic Control of Quality of Manufactured Product, Van Nostrand, 1931, Postulate 3 (archive.org scan). archive.org
- Toyota UK, TMUK's 25 Objects – Andon Cord, 2017-12-11. mag.toyota.co.uk
- Rigby, Bird, Convergent Contemporary Software Peer Review Practices, ESEC/FSE 2013. users.encs.concordia.ca (author's copy, quotes re-checked against it; the Microsoft Research mirror refuses automated fetches)
- Google Engineering Practices, Speed of Code Reviews. google.github.io
- Sadowski, Söderberg, Church, Sipko, Bacchelli, Modern Code Review: A Case Study at Google, ICSE-SEIP 2018, author's copy. sback.it
- Anthropic, Code Review, Claude Code documentation. code.claude.com
- Cursor, Bugbot documentation. cursor.com
- Paul Jialiang Wu, AI PR Review That Cannot Lie, 2026-09-08 (the reviewer's design and its first two runs). agentic-portfolio-lovat.vercel.app