Every memory layer claims it helps. We built a paired control experiment with deterministic oracles to test ours — including the results that hurt.
Benchmarks can't tell you if agent memory helps your team. A paired control can
We maintain nautilus-compass, an
open-source memory layer for coding agents (MCP tools: recall, ingest, drift
detection). Like everyone in this space, we claimed memory makes agents
better. Public benchmarks (LongMemEval and friends) measure QA over
Our first official benchmark runs — +14.2 points over a full-context baseline on LongMemEval at ~39× fewer tokens, plus the…
Architecture, decisions, and benchmarks from building an open-source memory evaluation framework for AI agents.
A practical, hype-free guide to benchmarking AI agent memory systems. Learn what metrics matter, how to design reproducible…
Everyone is building AI memory systems. But how do we know which ones actually work? As AI agents...
BEAM - the Benchmark for Evaluating Agent Memory - is a good benchmark because it tests memory the...
VetoBench measures the one thing memory benchmarks skip: when a task invites an approach your team already ruled out, does the…
