Your baseline scored 0.000? Before you publish the win, here is the checklist I now run, because a zero from a baseline is almost never a result. It is usually your harness.
This week I watched a builder on r/Rag do something rare: he benchmarked his own memory library against plain RAG expecting a win, got three null results, and published the confounds instead of the victory. Two of his bugs are so common, and so quiet, that I think every RAG or agent-memory benchmark should gate against them by construction. This post is the checklist that came out of that thread.
Bug 1: your arms have different context budgets
His first run compared a memory arm retrieving k=20 sentence-level hits against a BM25 arm returning whole sessions. Same "top-k" on paper. In characters, one arm got 1.3k of context and the other got 11.9k. BM25 looked dramatically better.
Once he matched the budget, accuracy went 0.28 to 0.59 for the memory arms and the ranking flipped. The original result was a budget difference wearing a granularity costume.
