A shorter, code-first version of a piece I first published in Towards AI on Medium.
If you've pointed an LLM at a pile of documents to extract a field — a total, a date, an amount — you know the uncomfortable part: it returns an answer for every document, all equally confident, and some are wrong. You can't check them by hand, because not having the answer is the whole reason you reached for a model. So the wrong ones ship silently.
I spent a couple of weeks on this and landed on something that helps, borrowed from an old idea called metamorphic testing.
The trick: test what shouldn't change
You usually know things that shouldn't change the answer:
