AI-Native Series · ANDON · Agentic PR Review and CI/CD · Episode 5 of 5
The Negative Test: a Gate Is Installed Only Once It Has Gone Red
Agentic PR review and CI/CD, episode 5, the last. In 1821 the United States Ordnance Department ordered two plugs for every musket barrel, one that must pass through and one that must not enter, because a gauge that only ever says "pass" tells you nothing. Most continuous-integration pipelines are full of gauges that have only ever said pass. This episode is the second N in ANDON, the negative test. It reports one repository's morning, when none of its six gates had ever been seen red, and its evening, when every gate carried a self-test with both plugs and a runner refused any gate without one; the three protections that had never fired; the replay pull request that is still waiting on a key; and the season's scoreboard.
1-minute takeaway — what you'll walk away with
What this is. Episode 5 of ANDON, a five-part series on agentic PR review and CI/CD, and its finale. This one is the rule negative test, built into a real repository as a self-test contract: every gate ships with a built-in fixture that must pass and a built-in fixture that must fail, a runner in CI enumerates every gate and goes red if any lacks the pair, and a replay pull request re-opens a known defect so that the AI reviewer must go red before it is trusted to go green.
Why it matters. For an engineer: a gate that has never fired is indistinguishable from a gate that cannot fire, and three of this repository's protections turned out to be the second kind. For a founder: Google measured that 84 percent of its pass-to-fail transitions were flakes, which is the other failure of the same gauge: a red nobody trusts is a gate nobody reads. For a student: the 1821 plugs are the whole idea in two pieces of steel.
What you can do after reading. Give every gate you rely on a self-test with both plugs. Add a runner that refuses a gate without one, and put the runner in the gate. Keep one replay of a real past defect open until your reviewer has caught it. And end the season the way it started: with the numbers that cannot yet be shown marked as not measured.
ANDON — the series spine
- A · Assess — The PR flood is measured, not felt episode 1
- N · Never silent — Jidoka for repositories episode 2
- D · Data, not prose — The rule that cannot drift episode 3
- O · Only in the envelope — Blast radius is a path list episode 4
- N · Negative test — A gate is installed only once it has gone red this episode
What this episode covers, and why the negative test comes last
Four episodes built a machine that decides: gates with no opinions, a third state for "could not run", rules that live in data, and an envelope that bounds what a bot may merge. Each of those is a claim about what a piece of code will do when the bad case arrives. This episode is about the only way to know that claim is true, which is to send the bad case through on purpose and watch the gate go red. It comes last because it is the test of everything before it, and because it is the one rule the repository had broken the longest without noticing.
The morning count is the whole argument. On 2026-09-09, before any of the flood gates from episode 1 were built, the repository had six gate scripts in its CI, and zero of them had a negative test. None had ever been shown to fire. The lessons file recorded three incidents in the preceding weeks with the same shape: a linter that was not installed and was skipped in silence, a pull-request template bot that the documentation described as present and that had never been wired, and a CI pipeline that had been silent for eleven days. In each case something described as a protection had never once done the thing protections do, and nobody could tell, because the green it produced looked exactly like the green a working gate produces.
So the rule for this episode: a gate is installed only once it has gone red on purpose, and the proof that it can go red ships with the gate.
The oldest version of the rule: one plug must pass, and one must not enter
The go/no-go gauge is older than the control chart, the checklist and the andon cord. Bomford's 1821 instructions for the Ordnance Department's musket inspection, as recorded in the Springfield Armory history, called for "two plugs for the bore diameter, with one to pass through the barrel and the other not to enter" [1]. Two plugs, not one, because a barrel that accepts the go plug tells you it is not too small, and only the no-go plug tells you it is not too large. A gauge with one plug is half a test. The dossier carries a caution with this date that this episode keeps: the plugs were ordered in 1821; the record does not show them in use, and the earliest documented pair is not the same claim as the earliest used pair.
The lineage on either side says the same thing in different metal. Honoré Blanc at Vincennes in July 1785, in Roser's reconstruction, "disassembled 50 locks, picked pieces for 25 locks at random, and … assembled them again" [2]: the demonstration was the test, and the point of choosing at random was that the assembler could not choose the pieces that would pass. And in 1896 Carl Edward Johansson, an armourer-inspector at a Swedish rifle factory, invented the combination gauge block set [3]: a reference that inspection tools could be checked against, so that the gauge itself had a gauge. Every one of these replaces "it looks fine" with a reference the thing must accept and a reference it must refuse, chosen so that the inspector cannot pick the specimens.
The kitchen picture from episode 1 has a last piece. The scale that weighs every plate is trusted because, before service, someone puts a known weight on it and reads the number, and then the checker is handed a plate deliberately fifty grams short and must send it back. A checker who has only ever passed plates might be waving everything through. The self-test is the known weight and the short plate, done every time the kitchen opens.
| Kitchen | Repository |
|---|---|
| The known weight on the scale before service, which must read right | The go plug: a built-in known-good fixture that every gate must pass |
| The plate deliberately fifty grams short, which the checker must send back | The no-go plug: a built-in known-bad fixture that every gate must fail |
| The opening checklist that says every scale was checked, or the kitchen does not open | The runner: enumerates every gate, runs its self-test, exits red if any gate lacks one or fails one; it is itself a gate |
| A dish sent back last month, cooked again to see whether the new cook catches it | The replay PR: a known past defect re-opened so the AI reviewer must go red |
The 30-year version: a red you cannot trust is a gate you stop reading
The software record adds the other failure of the same gauge. A gate that never fires is one problem; a gate that fires when nothing is wrong is the same problem from the other side, because both end with the gate being ignored. Google's 2017 measurement of its own continuous integration is the reference number: "84% of transitions from Pass -> Fail are from 'flaky' tests", and "Almost 16% of our 4.2M tests have some level of flakiness" [4]. A year earlier the same author had reported "about 1.5% of all test runs reporting a 'flaky' result." [5] The 2014 empirical study that named the causes found that "The top three categories of flaky tests are Async Wait, Concurrency, and Test Order Dependency." and that "Most flaky tests (78%) are flaky the first time they are written." [6] Google's follow-up found that "larger tests are more flaky" [7], which is the size gate from episode 4 arriving from another direction.
At Google's scale, with "more than 50,000 unique changes and running more than four billion individual test cases" a day [8], the answer to flakes is infrastructure. Chromium's commit queue states its retry rule plainly: "If the retry succeeds, then the failure is ignored." [9] For a small repository the answer is the same one the 1821 plugs give: know, for every gate, what it does on a known-bad input and on a known-good one, so that a red is a red and a green is a green. The self-test is what makes the gate's own signal trustworthy enough to act on without a retry.
One more historical note belongs here because it is about the origin of the checklist and it cuts against a comfortable story. The Model 299 crash of 1935, the accident usually credited with inventing the pre-flight checklist, happened because the gust locks were left engaged, and the aviation-safety account of it records that who left them engaged "is not clear." [10] The checklist was a response to not being able to tell. That is also what a self-test is for.
The build: both plugs on every gate, a runner, and a replay
1. The self-test contract
Every gate in the gate target and in CI, nine of them, now accepts a self-test flag. Run with it, the gate does not look at the repository at all. It constructs its own fixtures in memory: a router with a query that lacks the tenant filter, a commit message with an agent marker and no trace trailer, a policy with an exemption that has no reason, a set of workflow jobs containing a cancelled zero-step job, a rules table with one id changed. It runs its own check over each fixture and asserts the expected result in both directions: the bad fixture must produce a finding, and the good fixture must produce none. It prints how many checks it ran and exits non-zero if any assertion fails. The contract is written in the repository's lessons file in one sentence: a test in only one direction is the same as no test.
The counts, read from the runner's own output on the evening of 2026-09-09: the tenant-and-permission gate proves 2 checks can fire, the evidence-graph gate 4, the submission gate 10, the provenance gate 9, the CI-budget detector 4, the response clock 11, the AI reviewer 5, the golden case library 5, and the loop audit 5. Nine gates, every one of which has now been seen red, by itself, on every run of CI.
2. The runner is itself a gate
A self-test that a gate carries but nothing runs is prose again. So a runner holds the list of gates as data, runs each listed gate's self-test, and exits red if a listed gate is missing, lacks the flag, or fails its own assertions. The runner is in the local gate command and in CI, and it is the machine version of one of the repository's own review rules: the rule that says every gate must prove it can fire. The runner cannot notice a gate that was never listed; the list's completeness is pinned by the loop audit's structural criterion, which names the nine gates, and by the review rule. The loop audit itself was written with its self-test and added to the list in the same PR.
3. The detector that fired on the wrong thing
The other half of the 1821 rule showed up in the inventory of 2026-09-09. The repository has a detector that watches the Actions run history for the fingerprint of quota exhaustion, which on this plan looks like a job with zero steps. Run over the last sixty runs, it fired on one: a push to main on 2026-09-05 whose migration job had zero steps because the next push superseded it nine seconds in, under the concurrency rule. The detector's fingerprint matched a cancellation. The fix was not a threshold. The job carries its own conclusion, and a cancelled job says so; the detector now reads the conclusion and excludes cancelled, skipped and neutral jobs from the fingerprint, and its self-test includes a cancelled zero-step job that must not fire. A detector with a false positive is a no-go plug that sometimes enters, and a crew that has seen it enter once stops trusting it. The lessons file records that an earlier quota exhaustion had, in fact, gone unnoticed.
4. The replay PR
The AI reviewer has a self-test like the others, but its self-test exercises the parts that do not need a model: the evidence filter, the tier classifier, the brief renderer. The model's own judgment can only be tested by a real case. So the precursor article's build opened a pull request that reintroduces a known, already-fixed defect from the repository's history, the bypass that let a complaint be closed without passing its closure gate, and marked it "do not merge". The reviewer's rule for that defect exists; the PR is the no-go plug for it. On the day of writing the replay PR, open since 2026-09-06, carries one brief, unmeasured on a 401; the repository's eleven briefs in all say the same thing, because the key is rejected on every run. It stays open, deliberately, exempt from the response clock with a written reason, until the reviewer has gone red on it once. The article that promised to say which outcome arrived says: neither, yet.
Five whys: why a gate is installed only once it has gone red
| # | Why? | Because | Evidence |
|---|---|---|---|
| 1 | Why is a green from a gate that has never fired worthless? | Because a gate that cannot fire produces exactly the same green as a gate that found nothing, and nothing in the output distinguishes them. | Three protections in one repository had never fired; each was believed to be working until it was checked |
| 2 | Why does the proof have to ship with the gate rather than be done once? | Because a gate changes; the proof done at install time says nothing about the gate after its fifth edit, and the edit that breaks it will not announce itself. | The runner runs every listed self-test on every CI run; the loop audit shipped with its self-test and its place in the list in the same PR |
| 3 | Why both plugs and not just the no-go? | Because a gate that fires on everything is ignored as quickly as one that fires on nothing; the go plug is what makes the red mean something. | Google: 84 percent of pass-to-fail transitions were flakes [4]; the budget detector's cancelled-job false positive |
| 4 | Why must the runner itself be a gate? | Because a self-test nobody runs is a sentence in a file; episode 3's rule says a rule is what you can diff and episode 2's says a check that runs where the merge decision does not look has not been taken. | The runner is in the local gate command and in CI, and exits red on a listed gate that is missing, lacks the flag, or fails |
| 5 | Why keep a replay PR open and unmeasured? | Because the one gate whose judgment cannot be fixtured is the model's, and the only honest states for it are "went red on the known defect" and "has not yet"; closing the PR would convert "has not yet" into silence. | The replay PR's one brief, unmeasured with a 401; exempt from the response clock with a written reason |
The root, in one sentence: a gate's green is only evidence if the same gate has been seen red on a known-bad input since it last changed, and that seeing has to be part of the gate.
Build it in 30 minutes: the self-test contract
Thirty minutes is the contract's budget for a reader who copies it; it is not a time I measured.
| Element | Specification |
|---|---|
| Self-test flag | Every gate accepts --selftest. It builds its fixtures in memory, touches nothing in the repository, and asserts in both directions: known-bad ⇒ finding; known-good ⇒ no finding. |
| Output | One line: how many checks, all proven able to fire. Exit non-zero on any failed assertion. |
| Runner | A list of gate paths as data. For each: run the self-test; missing file, missing flag or non-zero exit ⇒ red. The runner is in the local gate target and in CI. |
| Detector fingerprints | Read the platform's own conclusion field before matching a shape; exclude cancelled, skipped and neutral. Put the false positive that taught you this into the self-test as a fixture that must not fire. |
| Replay PR | One PR that reintroduces a real, fixed defect, marked do-not-merge, exempt from the response clock with a written reason, closed only after the reviewer has gone red on it. |
| Invariant | known-bad ⇒ finding AND known-good ⇒ no finding, for every gate, on every run. |
| Stop rule | Any listed gate lacking the pair, or failing it, is red for the whole CI run; nothing merges past it. |
| Reporting | The loop audit's structural criterion counts gates with a self-test; its functional criterion runs the runner; both are read on every CI run. |
| Deliberately missing | No retry-on-flake for gates; no shared fixture library; no mocking of the platform beyond the in-memory fixtures. Each would be a place for the self-test to test something other than the gate. |
# The contract is two assertions per rule, inside the gate itself.
def selftest():
bad = fixture_with(agent_marker=True, trace_trailer=False) # must fire
good = fixture_with(agent_marker=True, trace_trailer=True) # must not
assert check(bad), "the gate never fires: not installed"
assert not check(good), "the gate fires on a good input: will be ignored"
print("selftest: 2 checks, all proven able to fire")
# The runner refuses a gate without one.
GATES = ["scripts/check_tenant_perm.py", "scripts/check_provenance.py", ...] # data
for g in GATES:
rc = run([python, g, "--selftest"])
if rc != 0: fail(f"{g}: no self-test, or it failed")
Same story, five exits
| Reader | The one decision | The one action |
|---|---|---|
| Student | A check you have never seen fail is not a check. | Take one thing that "always passes" and feed it a case that must fail. Watch what happens. |
| Engineer | Both plugs, in the gate, on every run. | Add --selftest to your most-trusted gate today, with one known-bad and one known-good fixture. |
| Founder | Count the gates that have ever been seen red. | Ask for that number this week. If it is zero, the badge on the README is decorative. |
| Executive | The 1821 plugs came in pairs for a reason. | Ask which of your controls has a documented negative test, and which are believed to work. |
| Investor | A team that keeps a replay PR open and unmeasured is reporting honestly. | Ask to see a gate going red on purpose, in the team's own CI history. |
Patterns, anti-patterns, and the first principle
Patterns that held. Fixtures in memory, both directions, inside the gate. A runner whose gate list is data and which is itself a gate. Conclusions read before shapes are matched. A false positive turned into a fixture. A replay PR held open and exempt until the model has gone red on it.
Anti-patterns this replaced. A linter that skipped silently when not installed. A bot documented as present and never wired. A detector whose fingerprint matched a cancellation. Six gates believed to work because they had never been seen not to.
The mechanism, named. A self-test converts "the gate works" from a belief about the past into a measurement taken on every run, and the runner converts "every gate has one" from a convention into a red.
The first principle, in one sentence. Trust a gate in proportion to how recently you watched it fail on purpose.
The season, in one table, and the scoreboard
ANDON was five rules, one repository, and one day of building against a four-horizon record. Here are the five, each with the letter, the episode's thesis, and the thing in the repository that is that rule, so that the series can be checked against the code rather than remembered.
| Letter | Rule | Thesis | In the repository |
|---|---|---|---|
| A | Assess with numbers | The flood is measured, not felt; the machine gate is the scale, not the chef | The dossier's live measurements; the machine gates that decide only what a script can decide |
| N | Never silent | A gate that could not run posts "unmeasured"; the response clock runs on a cron | The reviewer's third state; the SLA bot; the loop audit's P-criteria |
| D | Data, not prose | A rule is what you can diff; the docs are rendered from it and checked | Three JSON files; the renderer; the drift check proven red by hand |
| O | Only in the envelope | Where a bot may merge is a path list; deny wins; five conditions at once | The auto-merge envelope; the tier classifier; the provenance and size gates |
| N | Negative test | A gate is installed only once it has gone red on purpose, and the proof ships with it | The self-test on every gate; the runner; the replay PR |
| Season scoreboard | First morning, 2026-09-09 | Last evening, 2026-09-09 |
|---|---|---|
| Oldest open PR without a first response | 57 hours, three PRs | the same three; labelled, on the board, still unanswered |
| Gates with a self-test that proves they can go red | 0 of 6 | 9 of 9, enforced by a runner that is itself a gate |
| Agent-assisted commits since 1 September carrying a trace trailer | not measured | 11 of 21; required on new commits by the provenance gate |
| Loop-audit criteria unmeasured | no audit existed | 2 of 23: the AI has not measured a PR; Tier-1 has not merged one |
| AI briefs posted / silent | 3 / 0 | 11 / 0, all unmeasured on a rejected key |
Three of those rows are the season's honest failures. The first row is the one the season was named to move, and it did not: the three co-founder PRs went the whole season without a human response. The machine labelled them within the day; the maintainer never answered, and that is a person's action, not a setting. The other two are the same failure from two sides: a key that is rejected and a plan that has not reached the repository. Everything the series built classifies, posts, labels, audits and refuses correctly without them. Nothing the series built will claim to have measured a PR or merged one until it has. That was the first rule, and it is the last.
Reality mission
Episode 4 promised that by episode 5 every gate would have been proven able to go red by its own self-test, and that the replay PR would either have produced the reviewer's first measured finding or still be waiting on the key, with the article saying which. The first is done: nine of nine, on every CI run. The second is the second case: still waiting, one brief on the replay PR, unmeasured, like the other ten in the repository. The season's mission, which has no next episode to hold it, is three rows of the scoreboard. The first row is a person's mission, not a machine's: answer the three co-founder PRs, and the board will show the label come off. When the key is rotated, the replay PR's next brief will be measured; the reviewer will go red on the known defect or the series' central claim about it will have been wrong, and the dossier will record which. When the plan reaches the repository, one command installs the branch protection, and the next docs-only PR inside the envelope will either merge on its own or the audit will say why it did not.
Read next
Episode 5 of ANDON · Agentic PR Review and CI/CD, the last. Every number is scoped where it appears; the kitchen is a model, not a measurement. The self-tests, the runner, the detector and the replay PR exist in a private repository; their shapes are reproduced in the contract so nothing depends on access to it. The four-horizon dossier the season cites from is logged in the portfolio's plans directory.
References
- Raber, Malone, Gordon, Cooper, Conservative Innovators and Military Small Arms: Springfield Armory 1794–1968, NPS 1989/2006, p. 138 (Stubblefield and Lee to Bomford, 1821-12-04, NARA RG 156/21). npshistory.com
- Roser, 230 Years of Interchangeable Parts, AllAboutLean, 2015-07-08. allaboutlean.com
- Science Museum Group, Set of Johansson gauge blocks, c. 1900, object co59398. collection.sciencemuseumgroup.org.uk
- Micco, The State of Continuous Integration Testing at Google, ICST 2017 keynote. aster.or.jp
- Micco, Flaky Tests at Google and How We Mitigate Them, 2016-05-27. testing.googleblog.com
- Luo, Hariri, Eloussi, Marinov, An Empirical Analysis of Flaky Tests, FSE 2014. mir.cs.illinois.edu
- Listfield, Where do our flaky tests come from?, 2017-04-17. testing.googleblog.com
- Software Engineering at Google, chapter 23, Continuous Integration, 2020. abseil.io
- Chromium, Commit Queue documentation. chromium.googlesource.com
- Flight Safety Australia, One thing at a time: a brief history of the checklist, 2018-11-12. flightsafetyaustralia.com