Memory Is Not an Install — It's a Standing Measurement
On June 29, 2026, Microsoft Research released "Memora: A Harmonic Memory Representation" (ICML 2026) and published the code at github.com/microsoft/Memora. On long-conversation benchmarks it used up to 98% fewer tokens than dumping the full history into context, while scoring 86.3% LLM-judge accuracy on LoCoMo and 87.4% on LongMemEval — ahead of RAG, Mem0, Nemori, Zep, LangMem, and full-context reasoning. This post, though, is not an adoption guide for Memora; it takes the evaluation angle of what to measure and how to keep measuring whatever memory layer you bolt on. Attach memory once and forget it, and nobody notices the regression that shows up months later.
Retrieval Being Right Doesn't Mean the Answer Is
Memora's core insight is separating "what to store" from "how to retrieve." Your evaluation should split along the same two axes. A retrieval metric like retrieval@k tells you only whether the supporting passages were fetched — whether the final answer built from them is correct has to be scored separately by an LLM judge. Miss the band where retrieval recall sits at 90% but answer accuracy stalls in the low 70s, and you won't even know to look for the fault in generation rather than retrieval.
Trusting Public Benchmarks Alone Drifts From Production
LoCoMo and LongMemEval serve as coordinates for comparing memory performance against others, but datasets that have been public for a while carry contamination and overfitting risk. Memora's mid-80s scores on these two are a ranking within those benchmarks — no guarantee the same numbers hold on your support logs or long-running project histories. So keep the public benchmarks as a dashboard, but draw the acceptance bar for production judgment from your own domain golden set.
From Logging to Judging: A Roadmap for a Memory Evaluation Harness
In planning, nail down five target metrics. A reasonable starting line: memory answer accuracy (LLM-judge) at or above 85%, retrieval-evidence recall at or above 90%, context tokens per answer cut by at least 90% versus a full-history dump, stale recall — pulling outdated facts — at or below 3%, and judge-human agreement at or above 90%. To avoid the common trap of bragging about token savings while never measuring the specificity you lose, always place the savings rate and the accuracy figure side by side in one table.
Build the golden set from 30–50 sessions of your own domain conversations, labeling each question with both its supporting utterance and the expected answer. Deliberately mix in fact updates (say, asking for an old address after the user has moved) and time-dependent questions, because that is what surfaces stale recall. Run the public benchmarks weekly and the golden set as a per-commit regression, and the gap between a contaminated ranking and real usage shows up early.
Failure patterns fall into four branches. First, watching only retrieval@k and never LLM-judge accuracy, so you miss "retrieval right, answer wrong" entirely. Second, overfitting to LoCoMo and LongMemEval until the benchmark reads 87% while field support accuracy sits in the low 60s. Third, promoting a number like 98% token savings while never measuring the specificity loss that blurs fine-grained facts. Fourth, never calibrating the LLM judge itself, so a generous grader's scores get mistaken for capability.
Wire recovery branches per metric. When recall is high but accuracy is low, rework the generation prompt and the order in which evidence is injected; when accuracy and recall drop together, roll back the storage schema and chunking. When stale recall crosses its threshold, tune the fact-expiry policy and recency weighting; when judge-human agreement slips, exclude that release's automated scores from the trusted band and fall back to human grading.
Maintain judge calibration by having humans double-score a slice of the golden set each week and computing agreement against the LLM judge. If agreement falls below 90%, tighten the judging rubric, and mark the date you changed the judge prompt as a change line on the metrics dashboard so a sudden score shift can be traced to the model or the grader. Operational logs should carry session ID, retrieved evidence IDs, tokens per answer, the LLM-judge verdict, and a stale flag as required fields.
The improvement loop comes from treating the golden set as a living asset. Reinject the wrong answers and stale-retrieval cases caught in the field into the golden set weekly, and track the widening gap between public-benchmark and golden-set scores as its own metric. If that gap does not close, the harness fails to represent real usage — so suspect the golden-set composition first, not the benchmark.
Points You Can Apply Now
A memory layer is finished not when you attach it but when a measurement system stands behind it. Score retrieval metrics and LLM-judge accuracy separately, keep LoCoMo and LongMemEval as a dashboard while setting the acceptance bar on your own golden set at 85% accuracy, 90% recall, 3% stale recall, and 90% judge agreement, and always print the specificity loss next to the token-savings rate — then the next memory model can be compared on the very same harness.
References
Memora scales agent memory to boost long-horizon productivity — Microsoft Research
Microsoft unveils Memora to tackle AI agents' memory problem — InfoWorld