Originally published on hexisteme notes.
I recently froze a table of behavioral metrics computed from 281 of my own AI-agent sessions and ran it through two separate checks before I let myself trust any conclusion drawn from it. Both checks passed. Neither one verified the thing I actually needed to know — and both failures turned out to have the exact same shape.
The underlying table is observational, closer to a photograph of my own routing policy than a fact about the models sitting in it, and I've already written about that failure mode on its own. This post assumes that caveat and goes one layer past it: what happens after you've accepted it, built real checks anyway, and watched them both come back green.
Act one: a citation audit that passed 70 of 70
The corpus was 281 sessions, broken into 4,818 threads and 4,415 rows of behavioral metrics, frozen so nothing in it could shift under me mid-analysis. I had my own relaunched main model read the frozen table and write down its claims first, sealed before any outside model saw the data. Then I sent the identical table to six outside vendor families and asked each one to surface claims of its own. Only three of those six made it back cleanly — one hit a free-tier quota of zero, one threw an internal error, one had its response stream truncated mid-read. Between my own sealed read and the three outside reads that did return, four independent families ended up looking at the same numbers: my relaunched model, xAI's Grok, DeepSeek, and a Google open model.
