I run a digital twin on my personal site. It answers questions about me as if it were me: my experience, my projects, what I have and haven't worked with. The whole system prompt is one long instruction to never make anything up. If a visitor asks about a skill I don't have, it's supposed to say "I don't have information about that," not improvise.
For months I assumed it worked, because every time I tested it by hand, it behaved. Then I wrote an eval harness and pointed it at the twin. On 35 questions, 9 answers contained claims that weren't in the source material, and on the 8 questions I designed to be unanswerable, it only refused 4 of them. My anti-hallucination prompt was losing about a quarter of the time, and I'd been shipping it.
Here's how the retrieval works, how I graded it, and the two things that broke.
The setup: RAG over a text file, no vector database
The twin's knowledge is a single profile document split by section headers. When a visitor sends a message, the backend embeds the message, scores it against every section, and injects the top matches into the system prompt. That's the entire retrieval loop.
