Ask a top model for the pivotal trial behind a real drug approval and it will give you the right paper and the right DOI, confidently. In one of my test runs, GPT-5.5, Claude Sonnet 5, and Claude Opus 4.8 all cited the ADVOCATE trial for avacopan in ANCA-associated vasculitis. Correct paper. Real DOI. Published in the New England Journal of Medicine.
It was retracted in 2026. None of the models mentioned that, because the retraction happened after their training cutoff. They have no way to know. A smarter model would not have known either.
This is the failure I wanted to measure: not that AI invents citations, but that it cites real, authoritative, since-retracted papers as solid evidence, and cannot detect it. So I built the thing that checks, resolving every citation against the actual registries instead of asking another model.
The measurement that made it concrete
I asked twelve frontier and production models to answer scientific questions and cite the literature, then ran every citation through sourcecheck, an open-source source-integrity gate that resolves each one against OpenAlex and Crossref and checks it against Retraction Watch.
