Evaluation · September 9, 2026
Why Agent Memory Benchmarks Cannot Be Trusted
Every memory vendor publishes numbers where they win. What those runs leave uncontrolled, and the open harness we built to score our own system by the same bar.
By Shihab Shahriar Antor · Updated 2026-09-09
Agent memory became a category in about eighteen months. Mem0, Zep, Letta, Supermemory and a dozen others now sell some version of the same promise: your agent remembers across sessions, so it stops asking what you told it last week.
Every one of them publishes benchmark numbers. In almost every case the publisher wins. That is not evidence of dishonesty, and the explanation is more interesting than dishonesty would be.
Four things that decide the result before the queries run
| Choice | Why it decides the outcome | What an honest run does |
|---|---|---|
| Which suite | Systems are tuned for the retrieval shapes their customers hit. A suite whose questions are answerable by keyword match favours lexical retrieval; one built on paraphrase and multi-hop favours dense retrieval. Choosing the suite often chooses the winner. | Report on standard public suites, and say which shapes each one actually tests. |
| Which embedding model | This is the big one. Comparing a system running a strong embedder against a managed service using its own is a comparison of embedders wearing the costume of a comparison of systems. | Either hold the embedder constant across everything, or state plainly that the run is a system comparison and not a model-controlled one. |
| Whether the adapter is competent | A competitor integrated in an afternoon, with default settings and no use of its graph or document modes, will score badly. That number says more about the integration than the product. | Read the competitor's tuning documentation, use its intended mode, and treat a suspiciously low score as a bug in your adapter until proven otherwise. |
| Who judges correctness | Retrieval metrics are mechanical. Answer quality usually is not, so a model grades it against a rubric, and judge choice moves scores by several points. | Name the judge model and publish the rubric. Two judges is better than one; one named judge is far better than an unnamed one. |
None of these require bad faith. A team benchmarks the thing it built, on the workload it cares about, with the embedder it ships. Every one of those reasonable choices tilts the table.
What we built instead of a leaderboard
CH-Bench is the harness rather than the result. It scores any memory system on the same eval set through a four-method adapter, so the comparison is at least mechanically identical even when the configuration is not.
Three decisions in it are worth stealing whether or not you use it.
- 01
Score efficiency alongside accuracy
Every run reports latency p50 and p95 and token cost, not recall alone. A system that is four points better and three times slower is not better for an interactive agent, and an accuracy-only table hides that completely.
- 02
Measure retrieval and answer quality separately
Recall, MRR and nDCG say whether the right memory was found. Correctness and groundedness say whether the model then used it. These fail independently, and collapsing them into one score hides which half is broken.
- 03
Score abstention as a first-class outcome
A memory system that confidently answers a question it has no memory for is worse than one that says it does not know. Standard suites rarely test this, so we score it directly. It is the behaviour that matters most in production and gets measured least.
- 04
Add the questions real usage produces
Public suites test recall over a conversation. They do not test cross-project recall, or asking why a decision was made six months ago, or a fact that was updated three times. Those are the shapes people actually hit, so they get their own tracks.
- 05
Keep the harness dependency-free
Pure standard library, so it runs in CI and offline. A benchmark nobody can execute without provisioning four services is a benchmark nobody reproduces, and an unreproducible benchmark is marketing.
Reading somebody else's memory benchmark
If you are evaluating this category, most of the work is deciding which published numbers to ignore. Four questions get you most of the way, and they are quick.
Who ran it. Which suite, and does that suite reward the winner's retrieval style. Was the embedding model held constant. And is the harness runnable by you, on your data, without asking the vendor for anything.
That last one matters more than it sounds. Your documents are not LoCoMo. The only number that predicts how a memory system behaves on your workload is the one you produce on your workload, which is the entire reason we made the adapter interface four methods rather than an integration project.
Where this is going
The gaps in our own runs are the roadmap: hold the embedder constant across every system including the managed ones, use each competitor's intended mode rather than its defaults, add a second judge to bound judge variance, and grow the harder tracks where recall does not saturate. Until those are closed, we have a harness worth using and a set of numbers worth treating sceptically.
That is a less satisfying claim than a leaderboard. It is the one the evidence supports.
Questions
- What is an agent memory benchmark actually measuring?
- Two separable things. Retrieval quality asks whether the system surfaced the right stored memory, scored with recall@k, MRR and nDCG. Answer quality asks whether the model then produced a correct, grounded response, usually scored by a judge model against a rubric. They fail independently, so a single blended score tells you very little about what to fix.
- Why do memory vendors' benchmarks always favour the vendor?
- Mostly through legitimate choices rather than manipulation. The publisher picks the suite, ships its own embedding model, integrates competitors at whatever depth it has time for, and chooses the judge. Each decision is defensible on its own and each one tilts the result, which is why the identity of whoever ran the benchmark is the first thing to check.
- Should I trust LoCoMo scores?
- LoCoMo is a reasonable public suite for long conversational recall and it is far better than a vendor's private eval. It still only tests the shapes it contains. If your workload is cross-project recall, decision archaeology, or facts that get updated repeatedly, LoCoMo results will not predict your outcome well.
- How do I benchmark a memory system on my own data?
- Write an adapter with four methods, load your own documents and a question set with known answers, and score retrieval and answer quality separately while recording latency and token cost. CH-Bench is MIT licensed, runs on the Python standard library with no third-party dependencies, and exists to make that a day of work rather than a project.
- Is abstention really worth scoring?
- It is arguably the most important behaviour and the least measured. A memory system that answers confidently from nothing produces plausible fabrications attributed to your own past notes, which is a worse failure than admitting it has no memory of something. Most standard suites do not test it, so it has to be added deliberately.
The harness is on GitHub under MIT, including the adapters and the suite loaders. The system it was built to evaluate is Context Heavy, which is still in development and not yet released. The same scepticism in this piece applies to everything we publish about it.