The Filter That Refused My Own Paper

1. Measure the classics, don't admire them

Everyone quotes the Bitcoin whitepaper and Attention Is All You Need as models. Almost nobody measures them. Two minutes of pdftotext gives you something better than reverence:

My favourite hypothesis died on that table. "Classics build on old, settled work rather than fashionable preprints" fits Bitcoin perfectly, it is supported by real bibliometric research, and it flatters anyone who prefers first principles to trend-chasing. It would also have rejected the most-cited AI paper of the era. Reference age measures how fast a field moves, not how good a paper is.

A heuristic that fits one example is a coincidence.

What actually held was subtraction. Bitcoin removes the trusted third party. Attention removes recurrence and convolution — the title is the claim. Neither adds a part; both delete one and prove the deletion holds. That is why later work cites them instead of their sources rather than alongside them, which is precisely what bibliometricians call disruption.

2. Then twelve classics audited the filter

Two examples is enough to catch a coincidence and not enough to establish a rule. So I added ten more, measured the same way, spanning eight fields — Shannon, Turing, Einstein, Diffie–Hellman, Codd, Kahneman & Tversky, McCulloch & Pitts, AlexNet, Nash, Watson & Crick.

Three criteria survived everywhere, zero failures across all eight fields: does it name a specific open problem, does it hand over a mechanism a stranger can run, and does it say what would prove it wrong. Physics to biology to behavioural economics.

And the subtraction thesis got stronger — three of the greatest papers state the removal in their own words:

"The introduction of a 'luminiferous ether' will prove to be superfluous." — Einstein, 1905

"These semantic aspects of communication are irrelevant to the engineering problem." — Shannon, 1948

"there can be no general process for determining whether a given formula … is provable" — Turing, 1936

Then the expensive part. One of my criteria said a heavy hitter should be short — under 7,000 words. It failed eight of the twelve. Shannon's paper is 29,445 words, over four times the limit. Turing, Einstein, Diffie–Hellman, Codd and Kahneman & Tversky all blew through it too.

A rule that rejects Shannon, Turing and Einstein is not a quality bar. It was the average of a sample of two.

I deleted it, and recorded why, so it cannot quietly come back. A second criterion got demoted: subtraction is no longer required. AlexNet's abstract is purely additive — "considerably better than the previous state-of-the-art." Hand-engineered features died as a consequence of that paper, never as its claim. Subtraction measures how a paper is framed; displacement measures what it replaces. Only the second can be mandatory, or the filter throws out the decade's most influential vision paper for writing modestly.

The worst finding was about the instrument, not the criteria: the filter reported a clean pass while eight classics were failing inside it. It only printed subjects that fell below their floor, so a broken criterion stayed invisible. A filter you cannot audit across its whole corpus is one you are trusting, not testing.

3. Before: NOT YET

With the criteria repaired, the filter turned on my own preprint and refused it:

NOT YET   omi-v0.1: 4/5
  ✗ substitution-test: `displaces` is empty — name the work later papers
    would cite this INSTEAD OF, or accept that this joins a literature
    rather than redirecting it

The diagnosis was exact. My paper compiled my own principles into checks. That is an instance. A method binds a spec its author did not write, and I had never tried.

4. After: the method meets a document that never wanted to be checked

So I compiled two real, public commitment documents I had nothing to do with. The first was the OECD AI Principles — the most widely adopted AI values text in existence. Every clause got the same question: does this name something a third party could observe failing?

Four of ten clauses do. The other six cannot fail as written, and the measurement shows why. Across the 592 words of principles 1.1–1.5 there are zero instances of "must", zero of "shall", and eleven of "should" — plus twelve escape qualifiers: "appropriate to the context" three times, "consistent with the state of the art" three times, "where technically feasible", "as appropriate".

That is not a criticism of the OECD. Its instrument is a recommendation to governments, non-binding by design, agreed by consensus — it never set itself the bar I am measuring it against. That is exactly why it is the fair test: if the method says something useful about a document written with no intention of ever being gated, the method is doing work.

For contrast I compiled Anthropic's Responsible Scaling Policy, which scores 4 of 5 — because it chose to be checkable: 14 "must", 130 "will", and if-then structure with named thresholds. And the honest limit is enforced by a check rather than a footnote: any document scoring above 50% must state what a good score does not prove. Every threshold in that policy is set and adjudicated by the same company it binds. Checkable sentences are not external enforcement.

Nothing about the criteria changed to admit the paper.

That distinction is the whole point, because the easy failure of owning your own filter is negotiating with it. The bar did not move. The artifact did.

5. The best result was the one that cost me something

Compiling someone else's spec found two checks my own framework lacks. The OECD asks that systems "can be overridden, repaired, and/or decommissioned safely" — I have no gate for that; mine tests data exit, not system override. Anthropic commits to routine capability testing — I have nothing binding a testing cadence to a release.

Compiling my own manifesto could never have surfaced either. You cannot find the questions you did not think to ask by re-reading your own answers.

I did not implement them. A gate that passes because nothing declares the field it checks is a vacuous pass, and my own rules forbid that. They are named, owned, dated to v0.3, and a check now fails if any discovered gap lacks an owner — so it is work, not a note that expires quietly.

6. I also had to audit the auditor

I pointed an independent claim-checking tool from another of my repos at the paper. It returned "NOT READY — 14 over-claims." Alarming, until I read them: LaTeX table preamble, bibliography entries, and — best of all — the paper's own Limitations section, flagged as over-claiming for honestly limiting itself.

Exactly one of fourteen was real (an uncited characterisation of the sycophancy literature). The tool's own README had warned me its extractor is aggressive. Every super-tool in my playbook now carries a limit I measured myself, because a tool whose limit you have not measured will make your paper worse with authority.