AI-Native Series · Research Verification

My AI Caught the Fake Citations. Then It Named Its Own Blind Spots.

1-minute takeaway — what you'll walk away with

I pointed my own research-verification tool at two real, unpublished AI papers. It caught 8 fabricated citations and a headline metric with no results behind it — then it named the three things it still can't do. The strongest result was that it applied its own rule to itself.

I built a tool that decides whether a research claim is trustworthy. Then I pointed it at two real, unpublished papers — and made it grade itself. ~7 min.

My AI caught the fake citations, then named its own blind spots — what it caught vs. what it couldn't — The honest scoreboard: the wins on the left, the misses named on the right — not hidden.

The demo everyone runs is the one that lies to you

Every AI tool looks brilliant on the example its maker chose. The only test that means anything is the one where reality picks the input. So I took a tool I built — a research verifier whose whole job is to decide when an AI-written claim is trustworthy — and pointed it at two real, unpublished AI papers a colleague was drafting. With permission, I let it operate directly on the live project.

The rule the tool enforces is deliberately boring: a claim is "verified" only if its citation actually resolves and its evidence actually supports the strength of the claim. Otherwise it's demoted to ⚪ unproven — never quietly upgraded. Fixtures prove a thing runs. Real papers prove whether it's useful.

What it caught (in minutes)

Both drafts carried the exact failures that quietly erode trust in AI-assisted research:

8 fabricated citations. Placeholders like [cite: 611] that looked like references but pointed at nothing — a fact-shaped hole. The gate flags every citation that doesn't resolve; I replaced all eight with a visible [CITATION NEEDED] marker.
A headline metric with nothing behind it. Both abstracts advertised a "temporal coherence score of 0.92" — but there was no results table anywhere producing that number. The gate demoted it to ⚪ unproven. I rewrote it in-source as a target pending validation — I didn't fabricate a result, and I didn't silently delete someone's claim.
Over-reach on autonomy. Claims that "parallel agents" were doing high-autonomy work got auto-capped, because the ground truth there is contested — you don't get to claim self-driving where you can't cheaply check the answer.

Then the fixes were committed and pushed to the live project, each one a reviewable diff. Not a slideshow — a change in the source, reversible in history.

The claim-gate: a claim must have a citation that resolves and evidence that supports its tier, or it drops to unproven; plus the evidence ceiling ladder — The gate in one picture. A fabricated citation never resolves; a metric with no results is llm-synthetic. Both land at ⚪ — by rule, not by mood.

The mental model a 15-year-old can run

Give every claim a citation, then walk it up a short ladder. Does the citation resolve? No → unproven. Does the kind of evidence support the strength claimed? A number you observed in an experiment can be "verified"; a number an AI simply asserted cannot — it caps at unproven until something outside the model backs it. Only a claim that clears every rung earns the top tier. Everything else is labeled honestly, not thrown away.

That's the whole trick, and it's the same trick good science already uses. The tool just makes it mechanical — so "trust me" can't sneak past as "verified."

What it couldn't do — and why that's the real result

Here's the part most tool demos skip. The tool's own first principle is name your blind spots — so I made it apply that to itself. Three honest misses:

It can't read a paper into claims yet. I had to hand-extract the claims for it to judge. It gates claims; it doesn't yet ingest a document and produce them.
Its citation check is offline. It confirms a reference exists in the paper's own bibliography — it can't yet reach out and confirm a live source on the open web.
"Operating on the live site" worked through plumbing, not the tool. The reliable way in was the project's git backend, not the tool itself — a finding worth more than a polished success would have been.

So I did the obvious next thing: I built the missing piece — a manuscript reader that scans a draft for unwritten sections, fake-citation placeholders, and unsupported metrics, and feeds the candidate claims straight into the same gate. It shipped with tests the same day. The gap became a feature; the feature named its own next limitation (the extractor is still too eager). That loop — find the miss, close it, admit the new edge — is the point.

Why the misses are the win

A verifier you can't trust to be honest about itself is worse than no verifier — it launders confidence. The most important thing this run proved wasn't that it caught eight fake citations. It's that when I asked it to grade its own maker, it produced a scoreboard with a left column and a right column. Trust the artifact, not the label. The right column is where the trust actually comes from.

If you're building anything that judges quality — code review, research checks, an eval harness — the test isn't "does it pass its own demo." It's "does it tell you what it can't see." Build the one that fails out loud.

More in the AI-Native series

All of it lives in the Writing section on the home page.

Part of the AI-Native series. The verifier applied its own rule to itself — verify, don't vibe; name your blind spots. You own the Publish button.