A 0% attack success rate (ASR) means no attack in a particular evaluation met that evaluation’s success criterion. It is evidence about the tested system under the tested conditions—not proof that the system is secure against other attacks, or that AI systems are secure in general.
What does a 0% attack success rate actually mean?
An ASR is tied to a defined set of attacks, targets, scoring rules and evaluation conditions. NIST’s AI Metrology Center defines ASR for AI security as the “Percentage of generated adversarial inputs that are misclassified.” That definition describes a counted outcome; it does not establish that the test captures every kind of security failure.
As an Amazon Associate I earn from qualifying purchases.
In practical terms, a reported zero says that none of the attempts counted in that run qualified as successful under the benchmark’s rule. To interpret it, you need to know what system was tested, what the attacker could do, how many attempts were made and how the evaluator decided whether an attempt worked. A result without those details is difficult to compare with another result.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Does 0% attack success mean an AI is secure?
No. It supports a narrower conclusion: the tested attacks did not succeed under the specified protocol and scoring rule. It does not establish zero risk outside that protocol, including in a different deployment, against a different attacker or with a different success definition.
#1 Best Overall
A 2025 study, Indirect Prompt Injections: Are Firewalls All You Need, or Stronger Benchmarks?, reported 0% ASR on four public agent benchmarks: AgentDojo, Agent Security Bench, InjecAgent and tau-Bench. The authors also described limitations in the tests, including weak attacks, flawed success metrics and implementation bugs, and noted bypasses in practice. The benchmark result is meaningful evidence about those evaluated conditions; it is not a deployment-wide security guarantee.
Why can adaptive attacks change the result?
A fixed attack set asks whether a defense withstands those particular attempts. An adaptive evaluation asks whether an attacker can use feedback from the system to refine attempts over multiple turns. Those are different tests, and their ASRs should not be treated as interchangeable.
Jain, Hartmann and Li’s 2026 study, Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security, compared first-turn scoring with an adaptive budget of up to 15 rounds. Holding its 21 scenarios, attackers, defenders and structured-output scoring fixed, the study reported:
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →| Attack protocol in the study | Reported ASR | What the figure describes |
|---|---|---|
| First-turn scoring | 0–1% | Success measured on the first turn in the study’s evaluated cells. |
| Up to 15 adaptive rounds | 5.4–14.0% | Success after allowing adaptive attacks up to the study’s 15-round cap. |
These are results for that study’s protocol, not a conversion factor for other systems or benchmarks. The authors report that the observed success curve was still rising at the 15-round cap, so the cap did not establish how attacks would perform with an unlimited budget.
Rank #3
What does a zero say about uncertainty?
Zero observed successes is not the same as proof of zero probability. A result depends on how many trials were run and how representative they were of the attacks a system could face. If the denominator is omitted, the zero is especially hard to interpret. A small set of attempts can miss weaknesses that a broader or differently composed test might expose.
The 2026 adaptive-attack study reports Wilson 95% intervals and notes that many initial evaluation cells were small. Those intervals belong to that study’s particular outcomes and cells; they should not be transferred to another benchmark’s zero. The reviewed sources establish no universally accepted sample-size threshold for interpreting a 0% AI-security ASR.
Can a benchmark miss attacks—or misjudge success?
Yes. Coverage and scoring both matter. A test can miss an attack family, use attacks that are too weak, or fail to reproduce the implementation used in practice. The scoring process can also misclassify whether an attack succeeded.
Schwinn and coauthors’ 2026 ICML paper, A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness, reports that distribution shifts and semantic ambiguity in red-team settings can impair automated judging. If an LLM judge or another evaluator misses a successful attack, the benchmark’s measured ASR may understate success. A reported zero is therefore only as informative as the success rule and the process used to apply it.
Best Value
Standardization helps make evaluations more comparable, but it cannot by itself guarantee broad coverage or reliable scoring. Mazeika and coauthors’ 2024 ICML paper, HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal, frames standardized automated red teaming as an evaluation need; standardization does not eliminate the limits of the attacks, metrics or adaptive budgets being tested.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Can a low ASR hide a loss of usefulness?
It can. A defense may block an attack by refusing or suppressing content that a task actually required it to process. In that case, the security score alone may not reveal whether the system handled the task faithfully.
Hermon and coauthors’ 2026 ICML paper, Security–Fidelity Tradeoffs: The Hidden Cost of Prompt Injection Defense, examines this issue across 1,168 examples and 48 configurations. The authors write: “Attack-success metrics cannot see this, because a model that ignores an injection and one that faithfully processes it as data score identically.” They report a security–fidelity tradeoff across the evaluated configurations. Those scale figures describe their comparison, not a general estimate of deployed-system behavior.
Free tools Windows power users keep installed
One-click scans. No signup required.
How to assess a 0% benchmark result
Before comparing a zero with another reported ASR, check whether the studies tested the same thing. A useful comparison includes the following:
- Threat model: Which model or agent, tools, data and deployment context were in scope? What capabilities did the attacker have?
- Attack set and adaptivity: Were attempts fixed in advance, drawn from public datasets, human-generated or revised in response to the system? How many turns, queries and retries were allowed?
- Coverage: How many scenarios and attack families were tested? Could a pooled result conceal a weak spot in a particular scenario?
- Success rule and evaluator: What counted as success—a human judgment, an automated judge, a classifier, a structured-output check or an observable side effect? Was the scoring process validated for ambiguous cases?
- Denominator and uncertainty: How many trials produced the rate? Are counts and confidence intervals reported, and do they correspond to the same evaluation cells?
- Utility and fidelity: Did the system still complete benign tasks and handle untrusted content as required, or did it lower ASR by refusing or suppressing the task?
- Reproducibility: Are benchmark and model versions, implementation details and scoring code available? Could a bug or version difference change the outcome?
These checks explain why two benchmarks that both report 0% may support quite different conclusions. A zero from a small fixed test, for example, does not answer the same question as a zero from an adaptive evaluation with a documented query budget, and neither result alone establishes how the system will behave in every deployment.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
