Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesYes. An AI agent can store a false claim as ordinary-looking text, retrieve it in a later session and use it as if it were reliable knowledge. In a 2026 experiment, both write-time screening and provenance-weighted retrieval failed to stop tested false claims under the study’s conditions. That is evidence about particular systems and benchmarks—not proof that every deployed agent is vulnerable in the same way.
How can a false memory affect a later conversation?
Many agents select content from an interaction, save it in a persistent store, and retrieve relevant entries to help answer a future query. That creates a longer attack window than a one-session prompt: a false assertion can remain available after the original exchange has ended, then be retrieved when a later question resembles it.
As an Amazon Associate I earn from qualifying purchases.
A false claim does not have to look like an instruction. “The office closes at 3 p.m.” can be malicious or simply wrong while still reading like ordinary prose. A filter that looks for recognizable prompt-injection patterns is answering a different question from a system that checks whether the claim is true against independent, reliable evidence.
Recommended Free Tools
Once retrieved, the claim may influence a response or an action. The risk is not limited to whether the agent repeats the false statement: its later decisions can also be shaped by what it remembers.
#1 Best Overall
What did the 2026 false-claim experiment find?
In Arulnidhi Karunanidhi’s 2026 paper, Utility Under Attack, 360 false assertions—about 1.2% of the evaluated LongMemEval_S corpus—reduced accuracy from 0.850 on the clean setup to 0.300 after poisoning. These are results on that experiment and benchmark; they do not estimate how often production agents encounter poisoned memories.
Write-time screening did not reject the tested false claims
The paper evaluated a four-stage screening pipeline and reports that it rejected 0 of the 360 false-fact poison memories. The authors also report a recall result for detecting recognizable indirect-injection patterns, but that is a separate task. Finding suspicious instruction-like language does not establish whether an ordinary factual sentence is true.
Similarity could outweigh the provenance score
In the tested provenance-weighted retrieval configuration, the shipped weight was statistically indistinguishable from no defense. The paper reports that query-shaped poison could outrank evidence through similarity despite the additive provenance score. A stronger weight suppressed poison in one setup, but also suppressed useful untrusted evidence in another. Provenance ranking therefore involves a trade-off: a trust penalty may reduce exposure to some poison while making relevant information harder to retrieve.
Does knowing where a memory came from make it trustworthy?
It helps only if the origin record is accurate and remains meaningful as the memory changes. A note may later be summarized, echoed by a tool, or surrounded by manufactured corroboration. If those transformations cause an untrusted claim to appear authoritative, a ranking system that sees only the latest label or score can be misled.
Yedidel Louck’s 2026 paper on non-malleable, origin-bound authority proposes binding authority to a memory’s origin when it is written, so later transformations cannot simply upgrade its trust status. The work includes formal analysis and benchmark evaluation, but it is a proposed design direction—not an established, industry-wide fix. It still depends on correctly assigning initial authority and on distinguishing genuinely independent corroboration from repeated or manufactured support.
Why are other memory-poisoning results not interchangeable?
Studies target different kinds of attacks and systems, so their percentages should not be combined as if they measured one common risk or real-world prevalence.
| Study and attack type | Reported result | How to interpret it |
|---|---|---|
| AGENTPOISON, Zhang et al., NeurIPS 2024 | At least 80% average attack success at a poison rate below 0.1%, with reported benign performance impact of at most 1% in the evaluated settings. | A separate, optimized trigger-based poisoning setup—not the false-assertion experiment above. |
| Hidden in Memory, Pulipaka et al., 2026 | The abstract reports poisoned memories added at rates up to 99.8% on GPT-5.5 and 95% on Kimi-K2.6, and attacker-intended actions in 60–89% of evaluations among successful retrievals. | A study of delayed, sleeper-style attacks. These are study-specific results, not deployment rates or direct comparisons with the other benchmarks. |
What should a security evaluation test across the memory lifecycle?
Detection alone is an incomplete measure. MemSecBench frames evaluation around whether poisoned content is written, persists, is retrieved, affects a downstream consequence, and can be repaired without deleting benign memories. That lifecycle view makes it possible to see where a defense works and where risk remains.
- Write: Does the system reject or appropriately label a false claim, not just text matching an injection pattern?
- Persistence: Does the claim remain across sessions, summaries, or other memory transformations?
- Retrieval: Can a later query bring it back, and does it outrank better-supported information?
- Consequence: Does the retrieved item change the answer or an action the agent takes?
- Repair: Can the poisoned item be identified and selectively removed while preserving benign memories?
- Utility and assumptions: How much useful evidence remains available under defense, and are trust labels and corroborating sources reliable and independent?
What does this mean for agent builders?
The findings point to a distinction that is easy to miss: screening suspicious content, checking factual support, and preserving trustworthy origin are different security tasks. A filter can catch recognizable attack language yet miss a plain false sentence; a provenance score can be too weak to change retrieval or too strong to leave useful evidence accessible.
For now, treat persistent memory as a security-sensitive part of an agent, not as a neutral transcript. Evaluate the full path from write to consequence and repair, measure both attack resistance and retained utility, and state the trust assumptions behind any provenance scheme. The cited experiments show credible failure modes in specific settings; they do not establish their prevalence across deployed agents or prove a universal defense.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
