AI-Native Series · Reliability
I Fixed the Bug Three Times. The Bug Was Never the Problem.
1-minute takeaway — what you'll walk away with
I reported the same broken page to my AI agent three times; the third time in all caps. Each reply was true — fixed, merged, tests green — and the page stayed dead, because three different links in the delivery chain were broken and every layer above them kept reporting success. The lesson: "done" is a claim about the LAST link. Verify at the altitude the user experiences, or you're collecting your own receipts.
An autopsy of one afternoon: a wiring bug, a deploy platform that ghosted politely, and a feature that had never once run. ~6 min.
On August 2nd I told my AI agent that a private dashboard on this very portfolio showed "Not Found." It said it fixed the bug. I checked: Not Found. I reported it again. It fixed another bug. Not Found. The third time I typed WHY??? — three question marks, the international unit of stakeholder patience.
Here's the uncomfortable part: every one of those fixes was correct. The tests were green. The merges were real. And the page stayed dead through all of them, because I wasn't fighting one bug. I was fighting a chain — and a chain fails at exactly one link at a time, revealing the next break only after you fix the last one.
The mental model: a relay race where every runner grades their own leg. A fix has to travel — code → merge → build → deploy → serve → authenticate → the user's screen. Each runner in that relay will swear they ran a great leg, and they're usually right. But the race is only won when the baton crosses the last line, and nobody in the middle can see it. "Done" is a claim about the final handoff. Everything upstream is a runner's self-review.
Act 1: two halves, both correct, no handshake
The dashboard's API authenticates by reading one HTTP header. The dashboard's page fetched that API — and sent no headers at all. The API was correct: it rejected anonymous calls, as designed. The page was correct: it fetched its data, as designed. Nobody owned the sentence in between: the page must present the credential the API reads.
Google's SRE book has a line for this that I'd read years ago and apparently filed under "nice": "Note that in a multilayered system, one person's symptom is another person's cause" [1]. My symptom was a 404. The cause lived in the seam between two components that had each passed their own tests. Seams don't have test suites unless you write one — that's the entire argument for contract tests, where Fowler notes "[a] failure in any of these contract tests implies you need to update your test doubles, and probably your code" [2]. The contract was the thing nobody tested.
Act 2: the platform that ghosts politely
So we fixed the wiring, merged it, and told the user (me) it was done. Not Found.
The fix was on the main branch. The tests were green. What nobody knew: the hosting platform's free tier allows — verbatim from its own limits page — "100 times every 86400 seconds" [3], and an afternoon of enthusiastic small pull requests had spent all hundred. Every merge after that triggered nothing. No error in the repo. No red X. The deploy integration just quietly declined, like a mailbox with no bottom: it accepts your letters all day.
My agent kept telling me "merged ✓" the way a waiter says "coming right up" about a kitchen that's been closed since lunch. The claim was true. The claim was also useless, because "merged" is your own handwriting on your own receipt — the customer-facing question is "deployed and serving," and that question was never asked. The SRE book calls this the black-box/white-box split: black-box monitoring "is symptom-oriented and represents active—not predicted—problems: 'The system isn't working correctly, right now'" [1]. Every check we ran was white-box. The user was the only black-box monitor on duty, and the user was billing me in question marks.
Act 3: the feature that had never once run
Quota freed, deploy forced, page finally loads — and it's empty. The dashboard renders report cards that a CLI pushes to the site. That push, we now discovered, had never succeeded in its entire life: the CLI sent the credential under one header name; the site reads a different one. The feature had been tested — against the half of the path that doesn't need credentials. The authenticated half had zero production runs. Not "rarely run." Zero.
A code path that has never run in production isn't a feature. It's a rumor with unit tests.
Why three mentions? The Swiss cheese answer
James Reason's Swiss cheese model describes accidents in high-technology systems as trajectories through holes in multiple defensive layers — "some are engineered (alarms, physical barriers, automatic shutdowns, etc), others rely on people" [4]. It's usually invoked when the holes line up and disaster slips through. My afternoon was the inverse, and it's the version working engineers live daily: the slices were stacked so deep that fixing one hole just revealed the next slice's hole, one user complaint at a time. Defense in depth works both ways — it also provides failure in depth, each layer politely absorbing your fix and presenting a fresh "Not Found."
The chain only ends when someone verifies at the last link. The moment we finally drove a real browser, with the real owner session, against the real deployed page — the whole chain lit up: the wiring fix wasn't deployed, the deploy wasn't happening, the data pipe had never carried water.
Patterns / Anti-patterns
- Pattern — verify at the user's altitude. The definition of done for "fix the page" is a browser, logged in as the user, seeing the page. Not tests. Not a merge. Pixels.
- Pattern — every async handoff needs a positive acknowledgment. Merge → deploy is async. If you don't check the deploy actually started, silence and success are the same color.
- Pattern — the resurrection question. For any feature that crosses a boundary, ask: when did this last actually run, end to end, in production? If the honest answer is "never," you have a rumor, not a feature.
- Anti-pattern — collecting your own receipts. "Merged ✓," "tests green," "build passed" are all statements a system makes about itself. The user's screen is the only third-party audit.
- Anti-pattern — testing half a contract. If the authenticated path needs a secret your CI — the robot that runs your tests — doesn't have, your CI is testing a different feature that happens to share a name.
The mechanism, named: confirmation at the wrong altitude. Each layer confirms the thing it did, and human (and agent) brains hear a chorus of confirmations as one big "done."
The first principle, one sentence: a change exists only at the altitude the user experiences it; everything upstream is testimony from an interested party.
The receipts
Because an article about verification should be verifiable: three user reports; three distinct broken links (page→API credential, merge→build quota, CLI→API header name); one deploy platform limit of 100 per 86,400 seconds, fully spent; nineteen report cards pushed through the pipe that had carried zero — twenty, counting this incident's own report card. And my own accountability engine graded the turn 0.57 out of 1.0. The stored card (rc0020 in this repo's docs/reportcards/collection.json) records my claimed scores — four 1.0s — and the engine's grader caps each claim by the strength of its evidence when it reads the card back; run anyagent report roadmap over that file and it prints 0.57. My claim and my grade, permanently disagreeing in public — a number I then had to relay, verbatim, to the person I'd kept waiting. I recommend building systems that force you to say the quiet number out loud. It's the only reason this autopsy exists.
References
- Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (2016). Site Reliability Engineering, Ch. 6: Monitoring Distributed Systems. O'Reilly / Google. sre.google/sre-book/monitoring-distributed-systems
- Fowler, M. (2011). Contract Test. martinfowler.com/bliki/ContractTest.html
- Vercel. Limits — "Deployments per day (Hobby): You are able to deploy 100 times every 86400 seconds (1 day)." vercel.com/docs/limits
- Reason, J. (2000). Human error: models and management. BMJ, 320(7237), 768–770. pmc.ncbi.nlm.nih.gov/articles/PMC1117770
Related
Written the same day as the incident, from the receipts: the PRs, the deploy logs, the palace push output, and one user message with three question marks. — Paul Jialiang Wu · agentic-portfolio-lovat.vercel.app