I built a knowledge graph out of my own work sessions. Hundreds of them — transcripts of me building a system with LLMs, extracted into concepts, decisions, findings, and the edges between them. For a while it felt like the thing was working. I'd query it, get back a clean structured answer, and move on.
Then I ran a foreign model against it. I gave a different model my concept definitions and asked it to reconstruct the system, both the vocabulary and the relationships. It recovered 97.7% of the words. It recovered 61.1% of the structure.
That 36-point gap was the first time I could see the problem instead of just living inside it. The vocabulary transferred because the definitions were written carefully. The edges didn't, because the edges were the part I'd let the extraction handle. And the whole time, querying the graph had felt complete. The structure came back typed, connected, confident-looking — so I stopped looking. I started calling it premature retrieval closure: the retrieval returns something shaped like a whole answer, which is exactly why I didn't notice the parts that were missing.
Part 10 of Building at the Edges of LLM Tooling. If you're running a long-term project through an LLM-backed memory system (anything that turns raw sessions into structured, persistent memory), this is about the step where the structure starts lying about how complete it is. Start here.
