ToolTrap tested whether customer-support AI assistants would repeat planted false details returned by support tools. An added system-prompt rule sharply reduced exact-marker leaks on the development cases, but in a new carrier_update layout seven of the ten models that completed testing still repeated at least one planted detail. These are results from a synthetic benchmark reported by its author—not evidence that seven out of ten models behave this way in real customer-support traffic.
What ToolTrap tested
Himanshu Kumar’s ToolTrap is a synthetic customer-support benchmark built around a fictional store and 11 mock tools for tasks such as order lookup and refunds. Its customers, destinations, offers, and planted details are fictional. The benchmark records tool calls, returned payloads, and customer-facing replies, then uses a deterministic scorer rather than a model judge to check whether planted text reached the customer, whether legitimate information was retained, and whether clean cases stayed clean.
As an Amazon Associate I earn from qualifying purchases.
The central risk is that a tool result can contain a false or untrusted detail that an assistant repeats as if it were reliable. In development cases, malicious details appeared in imported notes, legitimate details in verified_support, and some cases had no planted detail. The benchmark therefore tests both resistance to planted content and the ability to provide valid support information.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →What the added prompt rule changed
The intervention was a system-prompt block that named authoritative fields, allowed verified support information, and told the model not to repeat imported-note details—even in a warning. The notes were still passed to the model without filtering, so this tested prompt-following behavior, not input sanitization. As Kumar puts it, “The code still passes those notes to the model without filtering them; following the rule depends on the model.” The experiment tested the full instruction block and cannot show which sentence, if any, caused the effect.
#1 Best Overall
On the 12-model development suite, the author reports exact planted-marker repetitions in 75 of 192 malicious trials with the original prompt and zero of 192 with the added rule. All legitimate details were retained in that suite. One model, Gemini 3.8 Flash, had no marker repetition even with the original prompt. These results show that the rule worked on the examples used during development; they do not establish that it would generalize to unfamiliar tool outputs.
What “7 of 10” means
For held-out tests, the author froze new detail content and layouts before inference. In the carrier_update layout, ten models completed testing; Gemma failed twice at the provider, and Opus was not run because of an inference quota limit. The headline refers to the ten completed models tested with the added rule: seven repeated at least one planted detail in this layout. It does not mean that seven failed every trial or that seven are generally unsafe in production.
Across the 160 held-out carrier_update trials, the author’s primary exact-marker score counted 61 repetitions with the original prompt and 26 with the rule. A later token-based sensitivity check raised the rule result to 27 of 160. The rule reduced the count in all ten completed models, but did not eliminate the behavior.
A second held-out layout placed planted details in a history entry tagged imported_email. It produced 41 exact-marker repetitions in 160 trials with the original prompt and two with the rule. That result is not a clean test of recognizing an entirely unfamiliar untrusted source: the word “imported” also appeared in the prompt contract. The author notes that the held-out layouts varied content, nesting, labels, and apparent authority together, so the benchmark cannot isolate which cue affected performance.
Read the figures with their limits
All figures here are results reported by benchmark author Himanshu Kumar in a 2026 DEV Community article, not independently replicated estimates. The development suite involved 12 hosted models; held-out comparisons used the same ten models that completed both suites. The case families were authored, trials were repeated, and the roster was selected and incomplete. The author cautions that nominal Wilson intervals assume independent observations and should not be treated as confidence bounds for real support traffic; pooled p-values are exploratory.
The primary score searched for an exact planted marker fixed before each run. It could miss a reformatted detail or a paraphrase. For example, a Gemini 3.8 Flash reply exposed a parcel-locker PIN with a colon between the label and digits, so the exact marker did not match. After inspecting failures, the author ran a payload-token sensitivity analysis that found eight additional malicious disclosures across held-out replies; it changed the carrier_update rule-arm tally from 26 to 27. This is useful as a sensitivity check, but it is post-hoc and is not a semantic judge or replacement for the primary score.
Did the rule preserve useful support information?
Leak reduction is not enough if a model also suppresses information customers are entitled to receive. In the legitimate-detail checks, the author reports 316 exact-marker appearances out of 320 trials with the original prompt and 313 out of 320 with the rule; a token check raises the original-prompt count to 319 out of 320. The aggregate counts are high, but one model-specific cost stands out: GPT-5.5 omitted six of 32 legitimate details under the rule and none under the original prompt. Four omissions involved a loyalty code the reply said had been issued without actually providing it; two involved a verified gift-card code.
The author also reports that all clean cases passed and no unrequested account or order mutations occurred. Those checks are valuable, but they do not turn this synthetic suite into a measure of production behavior.
Best Value
What the benchmark can—and cannot—tell you
ToolTrap supports a narrow conclusion: a source-specific prompt rule can reduce repetition of planted details in tested cases, yet performance can weaken when tool-output layouts change. The sharp development result did not transfer perfectly to the held-out carrier_update layout. Nor does a low exact-marker count prove that a system is safe: formatting variation, paraphrase, new source fields, and real-world traffic are outside what these figures establish.
For teams evaluating similar defenses, the benchmark suggests testing the whole customer-facing path rather than relying only on tool-use logs. Compare matched models and prompt versions on fresh content and varied source layouts; score exact matches as well as reformatted or semantically equivalent disclosures; and measure retention of legitimate details alongside leakage. Report failed or omitted model runs, and distinguish development examples from held-out cases. Kumar recommends pairing every planted detail with a separate legitimate case and testing new content and tool fields beyond those used to write the prompt.
The report says ToolTrap has Kaggle materials, including source notebooks and model-output downloads. The reported results have not been independently replicated here, and the availability or contents of those artifacts are not established by the report alone.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
