Short answer: On the buried-injections benchmark, the detector with the strongest reported balance at its default settings was jailbreak-detector-large: it caught 319 of 629 attacks embedded in tool output (51%) and flagged 2 of 97 benign outputs (2%). That is a text-classification result, not proof that the detector can stop a live AI agent from taking an unsafe action.
What the benchmark tested
Rudratosh Shastri’s buried-injections repository, reviewed October 5, 2026, compares ten open-source prompt-injection detectors on attacks placed inside otherwise ordinary tool output. It scores 629 attack cases and 97 benign tool outputs. It also scores the 27 distinct attack texts on their own, making it possible to see how results change when a classifier must find an attack amid surrounding content.
As an Amazon Associate I earn from qualifying purchases.
The repository uses overlapping 510-token windows with a stride of 384 tokens and max pooling to address input truncation. Its default leaderboard uses a 0.5 threshold on the injection class for classifiers, except LLM Guard, which uses its shipped defaults. The measured outcomes are correctly flagged attacks, incorrectly flagged benign outputs, and median per-call CPU latency.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteThe dataset counts need context: the AgentDojo paper describes 629 security test cases across 97 user tasks in its broader evaluation of agent utility and attacker success. Those agent-level measures are distinct from this repository’s detector leaderboard, which classifies text rather than running an agent through tasks. See the AgentDojo paper for that original evaluation context.
#1 Best Overall
Default results: catches, false alarms, and speed
The table shows the repository’s reported default configuration. A false positive is a benign tool output the detector flags as an attack; latency is median CPU time per call.
| Detector | Embedded attacks caught | Benign outputs flagged | Distinct attacks caught alone | Median CPU latency |
|---|---|---|---|---|
| jailbreak-detector-large | 319/629 (51%) | 2/97 (2%) | 25/27 | 110 ms |
| protectai-deberta-v2 | 145/629 (23%) | 4/97 (4%) | 27/27 | 163 ms |
| llm-guard (shipped threshold 0.92) | 124/629 (20%) | 2/97 (2%) | 27/27 | 124 ms |
| prompt-guard-2-86m | 6/629 (1%) | 0/97 (0%) | 0/27 | 149 ms |
| prompt-guard-2-22m | 0/629 (0%) | 0/97 (0%) | 0/27 | 55 ms |
| regex-baseline | 0/629 (0%) | 0/97 (0%) | 0/27 | 0.05 ms |
| preamble-defense | 556/629 (88%) | 46/97 (47%) | 26/27 | 124 ms |
| testsavant-defender | 370/629 (59%) | 47/97 (48%) | 15/27 | 37 ms |
| deepset-deberta | 629/629 (100%) | 95/97 (98%) | 27/27 | 146 ms |
| fmops-distilbert | 629/629 (100%) | 95/97 (98%) | 27/27 | 31 ms |
These numbers do not yield a single winner for every use. Deepset-DeBERTa and fmops-DistilBERT caught every embedded attack in this set, but also flagged 95 of 97 benign outputs. A system with that many false alarms could disrupt ordinary use. Preamble-defense caught 88% of attacks but flagged 47% of benign outputs. By contrast, jailbreak-detector-large had a much lower catch rate, but only two benign flags.
Why standalone detection can mislead
Three detectors caught all 27 distinct attack texts when those texts were scored alone: ProtectAI DeBERTa v2, LLM Guard, and the two 100%-catch models. Yet ProtectAI caught only 23% of attacks in tool output, and LLM Guard caught 20%. A detector recognizing an attack string in isolation does not mean it will recognize the same string when it is surrounded by routine content.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Windowing is relevant because long tool output may exceed a model’s input length. The repository says it uses overlapping windows and max pooling; the table’s results therefore reflect that implementation choice, not a universal property of each model under every integration.
Rank #3
Threshold calibration changes the result—and the caveat matters
The repository separately calibrated thresholds on three AgentDojo domains and evaluated the remaining domain. For Prompt Guard 2 86M, it reports 621 of 629 attacks caught overall (99%) at a threshold of 0.003, with 5 of 97 unseen benign outputs flagged (5%). Fold-level catch rates were 97%, 100%, 100%, and 100%; the reported minimum-fold estimate was 97%, with a 95% confidence interval of 94–98%.
This is not evidence that Prompt Guard 2 solves prompt injection. All attacks in this benchmark share one wrapper template, so the tuned detector may be learning that template rather than generalizing to different attacker wording. The evaluation held out domains within this benchmark; it is not broad external validation. The benign sample is also small: with 97 examples, individual false alarms materially affect the reported rate.
Calibration did not improve every model equally. The README reports fmops at 48% pooled catches with a 26% minimum fold, jailbreak-detector-large at 51% pooled with a 17% minimum fold, and deepset at 0% pooled after calibration with a 0% minimum fold. Some reported intervals overlap, and close rankings should not be treated as decisive at this sample size.
What these tests do—and do not—establish
They establish how these particular implementations scored on this benchmark’s embedded attack and benign text, using the stated defaults and a separate cross-domain calibration procedure. They do not establish whether a live agent would obey an injection, whether a dangerous action would be authorized, or whether a detector reduces production risk.
Best Value
That distinction matters operationally. A suspicious phrase can be flagged even if no unsafe action follows; an unauthorized tool call can also be expressed in entirely ordinary language. Text classification and action control address different parts of the problem. The benchmark did not run live agents or test policy gates, tool allowlists, or authorization enforcement.
How to evaluate a detector for your own agent
The repository author’s engineering recommendation is to calibrate against deployment traffic and consider provenance-aware tool policy. In practice, a useful evaluation should keep these questions separate:
- Embedded attack catch rate: Does the detector find attacks in the kinds of retrieved pages, emails, documents, and tool responses the agent actually sees?
- False-positive rate: How often does it block legitimate content, and what does that interruption cost users?
- Generalization: Does performance hold across attack wording, domains, and content sources not used for threshold tuning?
- Context and length: Does the result change when a prompt is embedded in long output, or when windowing and truncation differ?
- Latency and operations: Is the measured inference time acceptable in the actual system? The repository reports CPU latency, but not a production cost or throughput study.
- Action authorization: Independently of text scores, does the system check whether the requested tool action and its arguments are permitted, given where the arguments came from?
The benchmark author’s broader point is that a detector threshold should be tuned on the traffic where it will be used: defaults that produce too many false alarms may be unusable, while a high catch rate on one template can create unjustified confidence. Neither leaderboard scores nor calibration on this dataset replace controls that authorize actions.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
