How we measured a markdown knowledge graph as agent memory — lost to grep, rebuilt our search and editing primitives, retracted our own best number, and ended with a $4.50 curator whose store reads back at 96% of a hand-built ceiling in a single call.
The question
Every AI agent product eventually hits the same wall: the model forgets. Context windows end, sessions restart, and everything the agent learned about you evaporates. An industry has grown up around this — hosted "memory layers" that ingest your conversations, extract facts with an LLM, embed them into a vector store, and sell retrieval back to you by the API call.
IWE starts from a different premise. It is a local-first knowledge tool: your notes are plain markdown files in a directory, connected by links into a graph, readable by any editor and greppable by any tool. If that graph is a good memory substrate, an agent should be able to write its memories as linked markdown pages and read them back with graph-aware tools — no embeddings, no hosted pipeline, no per-retrieval meter. Memories you can open in your editor and correct by hand.
But "should be able to" is a hypothesis, not a feature. So we built a benchmark to test it — and the honest version of this story is that the benchmark spent most of its life telling us our hypothesis was wrong, each time pointing at exactly what to fix. This article is the story of that loop: measure, lose, build, measure again. By the end, the benchmark had reshaped IWE's search engine, motivated an entire block-level editing language, killed several of our own design rules, retracted its own most exciting number, and settled somewhere we did not predict: the cheapest curation model we could run, kept honest by mechanical guards rather than instructions, writing a store that a single retrieval call reads back at 96 percent of a hand-built ceiling — while the humblest baseline in the study, grep over the raw transcripts, still refuses to be beaten on accuracy alone. Both halves of that ending are the point.
