My agents were confidently wrong about the world, and I couldn't tell when. That's the part that got to me — not the wrongness, the confidence.
I run my one-person company as a fleet of about twenty AI agents — a content writer, a finance one, a researcher, a security officer, a handful more. They're good at the work I built them for. But every one of them shares a flaw I'd been papering over: when a task needs a fact about the world — how a tax threshold works, what a marketing framework actually says, how a platform bills — the model reaches into its training data and answers in the exact same self-assured tone whether it knows or is improvising. There is no tell. The guess and the fact wear the same face.
So this month I built the thing that was missing: a cited, fact-checked knowledgebase the agents have to read before they work, with a gate that keeps me from poisoning my own source of truth. Here's how it's built, the one rule that turned out to matter most, and the honest state of it — which is that I finished it days ago and have no idea yet whether it changes the work.
The job I was actually hiring this to do
Strip away my setup and the problem is one any solo operator using AI already has. You ask the model for something that depends on a real fact. It answers fluently. You either know enough to catch the error or you don't — and the whole reason you're asking is usually that you don't. The job I needed done wasn't "make my agents smarter." It was narrower and more honest: stop my AI from making things up in the one register where I can't catch it, and let me know which claims I can actually trust.
