Recommended Free Tools
A LoCoMo score is meaningful only alongside its test set, models, scoring rules and category coverage. A recent TrueMemory report gives EverMemOS 94.7% on single-hop questions and 94.5% overall, but says its lenient semantic scoring is not directly comparable with strict exact-match baselines. The benchmark name alone cannot tell you whether two published numbers measure the same thing—or establish that memory design caused their difference.
What does a LoCoMo score actually measure?
It measures how a system answered a particular set of questions under a particular evaluation protocol. “LoCoMo” identifies the benchmark, but that label does not, by itself, specify which questions were scored, which model generated answers, how correctness was judged, or how category results were combined.
As an Amazon Associate I earn from qualifying purchases.
Those details matter because each can change the reported result. A score based on semantic equivalence may accept an answer that strict exact-match scoring rejects. Excluding a question category changes the tested workload. Switching answer or judge models changes the conditions. A percentage without this metadata is not a complete comparison.
What is behind the reported 94.7%?
The accessible TrueMemory project report attributes 94.7% to EverMemOS on single-hop questions and reports 94.5% overall across its stated 1,540-question evaluation. The report describes a 10-conversation subset with four scored categories and excludes the adversarial category. It uses GPT-4.1-mini to answer, GPT-4o-mini to judge, and a majority vote across three judge runs. Its semantic-match rubric accepts answers with the same core topic or fact, including equivalent date formats. The report cautions that its lenient scoring makes its absolute results not directly comparable to published strict exact-match LoCoMo baselines. TrueMemory project benchmark report
#1 Best Overall
That 94.7% is a single-hop figure, not the report’s overall score. It also does not, on the evidence available, establish that the 94.7% in this article’s headline refers to EverMemOS. The matching number in the report is useful context, not proof of the headline figure’s attribution.
Why do published numbers differ?
Different questions or categories
Results can cover different conversation subsets or question counts, and an evaluation may omit a category. TrueMemory’s reported setup scores 1,540 questions across four categories while excluding adversarial questions. A result that includes a different selection is not automatically testing the same task, even if both papers call it LoCoMo.
Different answer and judge models
The answer model produces the response being evaluated; the judge model determines whether it counts as correct. TrueMemory reports GPT-4.1-mini for answering and GPT-4o-mini for judging. A separate Rovemark result card describing the Mem0-paper protocol reports 1,540 scored questions, excludes adversarial questions, and uses GPT-4o-mini as both answerer and judge. Rovemark LoCoMo result card
The figures from these setups should be treated as protocol-sensitive context, not as a controlled head-to-head ranking. Answer model, judge configuration, scoring details, system versions and execution conditions would need to be aligned before assigning a difference to one system.
Different scoring rules and metrics
Strict exact match and lenient semantic matching answer different questions. Exact match requires the output to meet a literal matching rule; semantic scoring can credit equivalent wording or formatting. A score under one rubric should not be read as though it were produced under the other.
Metrics also shape interpretation. The peer-reviewed MemoryOS paper reports category-level F1 and BLEU-1 results under GPT-4o-mini and Qwen2.5-3B conditions. These are metric- and model-specific results, not a single interchangeable percentage. MemoryOS paper, EMNLP 2025
Different system and retrieval configurations
Memory architecture is one possible source of performance differences, but systems may also differ in their versions and retrieval configurations. The TrueMemory authors say that, within their own comparison, “All 8 systems share the same answer model, judge, prompt, top-k, and scoring procedure. Only the retrieval layer differs.” That statement describes that report’s comparison only; it does not establish that protocols are matched across unrelated papers or vendor results.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteIs one 94.7% comparable to another?
Only when the underlying setups are sufficiently alike. Before treating two results as a ranking, check these details:
- System and version: Identify the exact system and version evaluated.
- Dataset and subset: Record the LoCoMo release, conversations and question count.
- Category coverage: Note which categories were included or excluded.
- Answer generation: Record the answer model and relevant generation settings.
- Judging: Record the judge model, prompt and number of judging runs.
- Correctness rule: Distinguish exact match from semantic scoring and note the metric used.
- Memory and retrieval: Record the configuration, including retrieval settings where reported.
- Provenance: Establish whether the result is vendor-reported, independently reproduced or a paper baseline.
If key fields differ or are unreported, present the figures as contextual references rather than a leaderboard comparison. “Not stated” is a useful description of missing methodology; it is not evidence that two setups match.
What can the scores tell us about memory design?
They can show performance under the protocol each source describes. Category-level scores and multiple metrics can also reveal distinctions that a single aggregate percentage hides. But the reviewed reports do not isolate or quantify how much of any particular gap comes from memory design rather than evaluation choices. It would overstate the evidence to assign a causal share to either factor without a controlled comparison.
For a stronger comparison, evaluate systems on the same question set, with the same answer and judge models, prompts, scoring rubric and execution conditions; publish category-level as well as aggregate results; and document each system’s memory and retrieval configuration. Even then, conclusions apply to the tested setup, not automatically to every benchmark version or deployment.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
