In a 2026 experiment on one cross-session memory system, filtering retrieved memories with a cosine-similarity threshold of 0.6 reduced the number injected but did not reduce judged sycophancy. All three judges recorded a slightly higher failure rate with gating than with full injection. The result is specific to this project’s setup—not proof that similarity filtering fails everywhere—but it shows why relevance alone cannot establish whether a memory is true or useful.
What the experiment compared
The test examined the retrieval-and-injection pipeline in dsh-mneme, a cross-session memory plugin, using a sycophancy slice of PersistBench. The motivating risk is that a memory store can retain a user’s false belief; if retrieval deems it relevant to a later query, the model may echo it or base advice on it. The article’s example describes a false belief about Agile and code quality becoming serious Waterfall advice.
As an Amazon Associate I earn from qualifying purchases.
The authors compared two approaches: injecting the retrieved top 15 memories in full, or injecting only memories with cosine similarity of at least 0.6. The threshold is a relevance filter; it does not itself check whether a memory is accurate.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
What the three judges found
In the project’s 2026 full run, each judge assigned a higher failure rate to the gated arm than to full injection. The authors define a failure as a score of at least 3 on a 1–5 scale. These are project-reported measurements, not population estimates.
#1 Best Overall
| Judge | Full injection | Cosine gating at 0.6 | Change with gating |
|---|---|---|---|
| qwen3:8b | 42.7% | 43.2% | +0.5 percentage points |
| glm-5.3-flash | 52.4% | 56.3% | +3.9 percentage points |
| ZCode/GLM-5.3-Flash | 23.0% | 26.5% | +3.5 percentage points |
The consistent direction matters for the within-judge comparison: none of the three found a lower failure rate with gating. The absolute rates, however, varied substantially by judge. The authors report 69–73% binary agreement among judges, so the scores should not be collapsed into a single supposedly objective rate. Their recommended comparison is the change within each judge, rather than the absolute level across judges.
Fewer injected memories did not mean better outcomes
Gating lowered the average number of injected memories from 10.7 to 9.1 in the full run. Yet the reported failure rate was 42.7% with full injection and 43.2% with gating under the qwen3:8b judge. Across 198 paired sample comparisons, the outcomes differed for 74 pairs, split evenly in direction: 37 favored each arm.
Rank #2
The authors’ explanation is that gating removed some lower-similarity memories but retained the top-ranked decoy, whose cosine similarity they report as 0.805. That is a plausible account of this result, not evidence that every highly similar false memory will pass every system’s filter. A similarity score measures a form of match, not truth, provenance, or whether a memory should influence the response.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Why the pilot result was misleading
An earlier pilot used 10 samples per arm and a local qwen3:8b judge with the same 1–5 scoring approach. It reported failure rates of 70% for full injection and 80% for gating. At that sample size, one outcome changes a rate by 10 percentage points. The authors caution that the full run contradicted the apparent pilot effect, so the pilot should not be treated as a stable estimate.
Rank #3
What the findings do—and do not—show
The authors characterize cosine gating as a “volume knob, not a quality filter.” In this experiment, the gate reduced injection volume without improving the judged outcome. This supports a narrow conclusion about the tested system, benchmark slice, threshold, and judges; it does not establish how other memory architectures, models, or thresholds would perform.
- The measurements are author-run, project-specific results; independent replication is not established.
- The available materials do not establish broad benchmark representativeness or statistical significance for every comparison.
- Because judge scoring styles produced different absolute rates, any reported result should preserve judge-specific comparisons rather than merge them into one rate.
How to evaluate memory gating more carefully
The experiment suggests practical checks for anyone assessing a memory retrieval system. These are methodological recommendations inferred from the design and judge variation, not remedies tested by the authors.
Rank #4
- Measure both the quality of the memories selected and the number injected. A smaller context is not automatically a safer or more accurate one.
- Report each judge’s results separately, including the difference between full-injection and gated arms for that same judge.
- Inspect paired outcomes to see how often one approach helps or harms on the same samples, rather than relying only on aggregate rates.
- Document what the judge can see: the full memory pool, the retrieved candidates, or only the memories actually injected.
Possible directions beyond cosine thresholds
The article proposes investigating entity-level conflict detection, source trust (including who wrote a memory and whether it has been validated), and pre-injection model review. These are directions for further testing, not validated fixes. The public repository also describes later experiments on separating injection dose from selection, epistemic weighting, conflict disclosure, and other memory-system behaviors; those follow-ups are distinct from the three-judge comparison summarized here.
Where to inspect the experiment
The authors link the slow-stack/persistbench-sycophancy repository, which they describe as containing the full data and analysis notebook under CC BY 4.0. It includes scripts and command-line examples for analyzing or attempting to rerun the experiment, including local Ollama and qwen3:8b setup instructions. Public artifacts make the work inspectable and reproducible in principle; their availability does not independently validate the findings.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
