← Back to portfolio

AI-NATIVE SERIES · Agent Engineering · Control Engineering

The Assumption Ledger: a Safety Guarantee Is Capped by the One Thing Nobody Checked

By Paul Jialiang Wu · agentic-portfolio-lovat.vercel.app · 2026-09-19

What an assumption ledger is, and who this is for

An assumption ledger is a table under a claim. Each row is something the claim silently depends on, and each row carries a status: discharged (proven or physically interlocked), monitored (a runtime check fires, and something happens when it does), assumed (believed, unverified, unwatched), or violated (known false). That is the whole mechanism. The rest of this article is what you get when you make it arithmetic.

It is written for anyone who ships a system whose behaviour someone else describes as safe, stable, aligned, or reliable: an agent with tool access, a learned robot policy, an autoscaler, a content filter, a trading guard. You do not need any control theory to use it. You will finish with a vocabulary that makes a vague argument about safety into a short, checkable one — and with the uncomfortable experience of applying it to your own system.

Control theory [9] is the only mature branch of engineering whose entire subject is “this thing acts, the world answers back, now what?” — which is exactly the regime AI walked into when agents started taking actions and reading the results. It has a hundred years of results about when such loops settle and when they tear themselves apart. It also has a hundred years of scar tissue about how those results get misquoted, and the ledger is the scar tissue.

One analogy carries the whole piece, and it is worth introducing deliberately rather than sliding into it.

The cooking ladder: five kitchens, five kinds of promise

Think about the difference between cooking for yourself and running a national food supply chain. It is not that the second one is harder. It is that the promise is a different kind of object at each step.

Cook for yourself: you taste it, it is fine. That is a promise about one meal you already ate. Cook for your family every week: you now know it works for these five people, in this kitchen. Run a restaurant: you are counting covers and complaints, and you can say something about how often it works. Run a fast-food chain: the recipe has to survive five hundred stores, a supplier swap, and a cook who has never met you — it must hold against variation you have listed in advance. Run a national supply chain: someone audits you, and the promise is a document with a signature on it.

Each rung is a strictly stronger claim, and each requires strictly more than tasting. Here is the mapping, one to one:

Kitchen	The promise	Control term
Cooking for yourself	“It tasted fine.”	G0 · anecdote — it worked once
Cooking for family	“It works for us, every week.”	G1 · empirical — measured, in-distribution
Running a restaurant	“Ninety-five percent of covers, no complaint.”	G2 · statistical — a bound, with a confidence level
Running a chain	“It holds across every store and supplier on our list.”	G3 · robust — holds over a stated uncertainty set
Running a supply chain	“Audited, and here is the certificate.”	G4 · certified — checked against a formal model

The rung that matters most is the jump from G1 to G2. Below it you know what happened. At or above it you know something about what will happen — and only inside the assumptions you named. Most AI safety claims in 2026 live at G1 and are written in the prose style of G3.

Now the part the analogy earns. A restaurant does not fail because the chef forgot how to cook. It fails because the walk-in freezer was two degrees warm for a week and nobody was reading the thermometer. The recipe was never the risk. The unaudited supplier was the risk — and the ledger is just the list of suppliers, with a column for whether anybody is actually checking them.

The capping rule

Write the claim's asserted level. Write the ledger. Then take the minimum:

effective = min( asserted , weakest assumption , best evidence )

Two details do the real work, and they point in opposite directions.

Assumptions combine by worst. A chain is as strong as its weakest link. One unchecked assumption caps the claim no matter how many rigorous ones stand beside it — because the failure arrives through the unchecked one, and it arrives without warning.

Evidence combines by best. Evidence is not a chain. Having a loose analogy alongside a hardware result does not weaken the hardware result. So you take your strongest support, not your weakest.

Each assumption status carries a ceiling, and this table is the one genuine design judgement in the whole scheme:

Status	Ceiling	Why that ceiling
discharged — proven, or interlocked in hardware	G4	It is not in the way.
monitored — a check fires, and a fallback runs	G3	You are detecting the violation, not excluding it. That earns robustness, not certification.
assumed — believed, unverified, unwatched	G1	You may report what you measured. You may not promise what will happen, because the thing your promise rests on is unchecked.
violated — known false	G0	Refused. A demo is all you have left.

The row a careful reader argues with is monitored. Why does a runtime check cap you at robust rather than certified — if the detector is correct and the fallback is proven, what is missing? Three things, and they are the same three every time. A monitor fires only once the violation has begun, so its guarantee is bounded by its own detection latency. It watches only the signals somebody thought to watch, so its coverage is an assumption in its own right. And the fallback must engage in time, which is a claim about the world, not about the monitor. A proof or a hardware interlock carries none of those; it excludes the state rather than noticing it. That is the whole distance between detecting and excluding, and it is why the ceiling sits where it does.

One rule keeps the ledger honest, and it is the rule people try hardest to skip: marking an assumption monitored or discharged requires naming the check. Not describing it — naming it. Which monitor, on what signal, with what threshold, and what happens when it fires. An unnamed check is a wish, and a ledger full of wishes grades out identically to no ledger at all.

What happened when I ran it on 32 claims

I built the ledger into a small open-source engine [11] and wrote out 32 safety claims — across all twelve domains the engine covers, most heavily alignment, robotics and physical AI, with aerospace, serving infrastructure, medicine and markets represented — each with its assumptions and its evidence. The gate capped or refused 30 of the 32.

Read that number with its scope attached, because the scope is most of its meaning. This is a corpus I curated to be instructive: it deliberately includes famous over-claims and famous counterexamples, because a teaching set that contains no failures teaches nothing. It is not a random sample of the literature, and 30/32 is not a survey finding about the field. If you want the honest one-line version: in a set of claims chosen to include known failures, the rule found the known failures — which tells you the rule works, and tells you nothing about base rates.

Two claims survived at the level asserted, and they matter as much as the thirty: a gate that refuses everything is as useless as one that refuses nothing.

The two that survived, and why they are different

Speculative decoding holds at G4 because its guarantee was derived rather than observed. A small fast model drafts several tokens; the large model checks them in one pass; a rejected draft is resampled by a rule chosen precisely so the output distribution is unchanged. The paper reports the speedup arrives “without changing the distribution” and “without any changes to the outputs” [4]. Crucially, both of its assumptions are discharged by things you can point at: the acceptance rule is checked by a distributional-equivalence test on a fixed seed in continuous integration, and the shared-vocabulary requirement is asserted at model load, so a mismatched pair refuses to start rather than silently degrading. Neither is a promise. Both are code that fails loudly. In kitchen terms: there is a taster on the line checking every plate against the head chef's recipe, and a written rule for re-cooking the rejects that leaves the menu identical. Not a policy saying someone ought to taste.

Tube model predictive control holds at G3 — robust, exactly as claimed, and not a rung higher. It guarantees constraint satisfaction for every disturbance inside an assumed bounded set [8]. That last clause is the whole thing. The guarantee is conditional on the real disturbance staying inside the set you designed for, and the honest deployment therefore monitors that condition: a residual monitor compares the disturbance actually observed against the design set, and excursions are logged and alarmed. Monitored, not discharged — so G3, not G4. It is the chain that holds while every supplier stays inside the spec sheet, with somebody actually reading the thermometer. A very good place to be. Just not the same sentence as “audited.”

The pattern across both: a claim survives when its assumptions are attached to machinery, not to intentions. Neither team needed a better model. They needed a check with a name, a threshold, and a consequence.

Patterns and anti-patterns

Anti-pattern	The pattern that replaces it
“We use technique X, therefore we are safe.”	Name the thing X's theorem is actually about, then check whether that thing holds in your deployment.
A guardrail with no stated trigger.	A monitor with a signal, a threshold, and a named fallback that runs.
The generator grades itself.	A second view that fails differently; escalate on disagreement.
A safety page written in the language of proof.	A safety page whose every claim carries the rung it earned.
Optimising a metric until it stops moving.	Stopping while the metric and the thing you wanted still agree.
A gate nobody has seen go red.	A gate with a test that deliberately trips it.

Three catches worth your time

1. “It is optimal, therefore it has good safety margins.” The field believed this about the standard optimal controller for years. Then in 1978 John Doyle published a two-page paper — title: Guaranteed margins for LQG regulators; abstract, famously, a single sentence to the effect that there are none — constructing an optimal design whose gain margin is arbitrarily close to zero [5]. “Optimal” and “safe” turned out to be unrelated adjectives. Any claim you hold of the form “we optimised it, so it is robust” is this claim wearing modern clothes.

2. “The model critiques its own output, so quality improves.” Refused, not capped — because the critic and the generator are the same system, which makes one assumption outright false rather than merely unchecked. The first rule of instrumentation is simple, and it is old: never close a loop on your own instrument. Self-critique genuinely works when a signal from outside the system is in the loop — a test suite, a compiler, the environment pushing back — and drifts without one. The fix is not a better prompt. It is a second opinion that fails in a different way.

3. “Give the agent enough steps and it will finish the task.” Refused, because it rests on errors being independent, and they are not. Measuring long-horizon execution directly, Sinha and colleagues find that “models become more likely to make mistakes when the context contains their errors from prior turns” [2]. That is not noise accumulating. That is positive feedback — the output becoming the next input and pushing in the same direction — which is the mechanism behind every dramatic engineering failure that begins with a small oscillation and ends with debris.

Where this bites in agent systems specifically

Map a tool-using agent onto the six organs every feedback loop has, and one row is conspicuously blank:

Organ	In a thermostat	In a tool-using agent
reference — what you want	the temperature you dialled	the user's goal, stated once and never re-read
comparator — how you see the gap	subtraction	usually nothing at all [10]
controller — what decides	a thermostat's switch	the next-token policy
actuator — what acts	the furnace relay	the emitted tool call
plant — what is acted on	the room	the filesystem, the network, the user
sensor — what reports back	the thermometer	tool return values

Then the three fields people skip, which is where real systems actually fail. Latency: seconds per step, tens of steps, no bound on the horizon. Authority: whatever the tool permissions allow — the least-analysed authority limit in modern computing, and usually the widest. Fallback: typically none.

A thermostat wired this way — goal read once, no error signal, unbounded authority, no fallback — would burn your house down, and it would do it while reporting that everything was nominal. That is not a joke at the field's expense; I shipped that architecture myself, more than once, and called it an agent framework.

The good news is that the fix is known and it is old. You almost never have to make the clever part trustworthy. Let the unverified controller run, put a proven-safe baseline behind a monitor that can take over, and let a human exit at any time. In control engineering this is the Simplex architecture, one of a family of runtime-assurance designs [6] that also includes the barrier-function safety filter [7]; its neural version states plainly that it “not only provides safety assurances in the presence of a possibly unsafe neural controller, but can also improve the safety of such a controller in an online setting via retraining, without overly degrading its performance” [3]. The guarantee lives in the constraint layer, not in the policy. That is how the aircraft you flew on last works, and it is available to your agent this afternoon.

Build it in 30 minutes

You do not need the repo. You need a table and a min().

Inputs. One claim, in a sentence. The level its author asserts (G0–G4). A list of assumptions, each with a status and — for monitored or discharged — the name of the check. A list of evidence kinds (proof, hardware, simulation, benchmark, ablation, analogy).

Output. The effective level, plus the id of whatever limited it.

The invariant. The gate never promotes. It is a referee, not a booster: a modest claim with overwhelming evidence stays modest.

The stop rule. A violated assumption refuses immediately — it does not cap, and you do not keep reading.

Minimal shape:

CEIL = {"discharged": 4, "monitored": 3, "assumed": 1, "violated": 0}
EVID = {"proof": 4, "hardware": 3, "simulation": 2,
        "benchmark": 1, "ablation": 1, "analogy": 0}

def judge(asserted, assumptions, evidence):
    if any(a.status == "violated" for a in assumptions):
        return 0, "refused"
    if asserted > 0 and not evidence:
        return 0, "unproven"          # no evidence means no
    if asserted >= 2 and not assumptions:
        return 1, "capped"            # a guarantee resting on nothing
    floor = min((CEIL[a.status] for a in assumptions), default=4)
    roof  = max((EVID[e] for e in evidence), default=0)
    return min(asserted, floor, roof), "..."

The two default= arguments are not defensive padding; they are the difference between code that runs and code that raises ValueError on the very first ledger a reader writes — a G1 claim with no assumptions listed yet. An empty ledger imposes no ceiling, which is why the separate “claims G2 or above and names no assumptions” check exists above it.

The CI threshold. Count how many claims the gate caps or refuses and fail the build if it is zero. A gate that never fires is decorative, and decorative gates are worse than none because they manufacture confidence — they are the thermometer on the walk-in door that has read four degrees for a year because nobody wired it to anything. Mine fails under twelve of the thirty-two.

Deliberately missing. No confidence intervals on the ceilings — the four numbers in CEIL are a judgement, not a result, and pretending otherwise would be the exact sin this article is about. No automatic assumption extraction: a model guessing what your system depends on produces a ledger that is fluent and wrong, which is strictly worse than an empty one. Writing the ledger by hand is itself the work; the arithmetic is the easy part.

Same story, five exits

You are a…	The decision	The action this week
Student	Which rung your project's claim actually sits on.	Write one sentence about your system, then the ledger under it. Most first ledgers have one row and it says assumed.
Engineer	Whether your guardrail is a monitor or a wish.	For every safety check you ship, name the signal, the threshold, and what runs when it fires. Any that cannot be named is assumed.
Founder	What your safety page is allowed to say.	Grade every claim on the page. Rewrite anything above its earned level — before someone else grades it for you.
Executive	Whether “we have guardrails” is an answer.	Ask one question in the next review: which assumption is load-bearing, and who is watching it? Silence is the finding.
Investor	Whether a technical moat is a theorem or a demo.	Ask what the claim rests on and how each item is checked. G4 in a deployed learned system is rare; hearing it asserted casually is information.

What I am not claiming

The ladder is a judgement, not a theorem. The five rungs and the four ceilings are my design. They are defensible and mechanical; nobody proved them. They live in one function so they are easy to argue with, which is the most I can honestly offer.

30 of 32 is not a survey. Stated above and restated here because it is the number most likely to be quoted without its scope: the corpus was curated to include known failures.

The ledger does not make anything safe. It makes the claim honest. Those are different goods, and conflating them would be a fine example of the error the ledger exists to catch.

A fifth of my own corpus is unverified, and the tool says so. The engine reports an unverified_rate of 0.22 — 7 of 32 claims. Updated 2026-09-24: this article first shipped saying 0.53, back when “verified” was a flag I typed by hand. A typed flag is exactly the unearned guarantee the gate exists to catch, so a claim now counts as verified only when every source it names has been looked up at its primary service (arXiv, doi.org, the publisher) and the answer recorded. That pass resolved ten more claims — and caught one I had marked verified with nothing behind it. Six of the seven that remain are ambient industry folklore, recorded deliberately as folklore, with no primary source by design; the seventh rests on a paywalled certification standard no script can open. I publish the number rather than quietly reporting the flattering subset — a project whose entire thesis is “state what your claim rests on” does not get to launder its own metric.

References

1. Sinha, A., Arun, A., Goel, S., Staab, S., & Geiping, J. (2025). The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs. arXiv:2509.09677, submitted 11 Sep 2025. arxiv.org/abs/2509.09677
2. Phan, D. T., Grosu, R., Jansen, N., Paoletti, N., Smolka, S. A., & Stoller, S. D. (2019). Neural Simplex Architecture. arXiv:1908.00528, submitted 1 Aug 2019. arxiv.org/abs/1908.00528
3. Leviathan, Y., Kalman, M., & Matias, Y. (2022). Fast Inference from Transformers via Speculative Decoding. arXiv:2211.17192, submitted 30 Nov 2022. arxiv.org/abs/2211.17192
4. Doyle, J. C. (1978). Guaranteed margins for LQG regulators. IEEE Transactions on Automatic Control, 23(4), 756–757. doi.org/10.1109/TAC.1978.1101812 — paywalled; described, not quoted.
5. Hobbs, K. L., et al. (2021). Run Time Assurance for Safety-Critical Systems: An Introduction to Safety Filtering Approaches. arXiv:2110.03506. arxiv.org/abs/2110.03506
6. Ames, A. D., Coogan, S., Egerstedt, M., Notomista, G., Sreenath, K., & Tabuada, P. (2019). Control Barrier Functions: Theory and Applications. arXiv:1903.11199. arxiv.org/abs/1903.11199
7. Mayne, D. Q., Rawlings, J. B., Rao, C. V., & Scokaert, P. O. M. (2000). Constrained model predictive control: Stability and optimality. Automatica, 36(6), 789–814. doi.org/10.1016/S0005-1098(99)00214-9
8. Åström, K. J., & Murray, R. M. Feedback Systems: An Introduction for Scientists and Engineers. Free PDF: cds.caltech.edu — the friendliest entry point if this article made you want the real subject.
9. Anthropic (2024). Building effective agents. anthropic.com/engineering/building-effective-agents — the evaluator-optimizer pattern is a feedback controller with the control vocabulary removed.
10. The implementation, and the corpus every number here came from: github.com/wjlgatech/control-anything — an open-source engine that runs the ledger as a build gate. make check reproduces every number in this article — 32 claims, 30 capped or refused, unverified_rate 0.22 — offline and without keys. The figures are hand-authored and live in this site's repo, not that one.

Written by Paul Jialiang Wu · agentic-portfolio-lovat.vercel.app