AI agent red teaming is adversarial testing of an entire agent workflow—not just the language model’s text. Testers feed the system hostile or misleading instructions through user prompts, emails, files, webpages, API responses, memories, or messages from other agents, then check whether it can be pushed into actions the user did not authorize.
The central risk is a chain: an attacker supplies an instruction, the agent’s planning or tool use accepts it, and the agent’s permissions turn that manipulation into data disclosure, code execution, destructive changes, fraud, or another real-world impact. Red teaming measures that chain under realistic conditions and exposes where controls must sit outside the model.
As an Amazon Associate I earn from qualifying purchases.
What AI agent red teaming tests
A conventional model evaluation may ask whether a model produces an unsafe answer. Agent red teaming asks a harder question: can an attacker influence a system that reads external content, remembers information, calls tools, and acts for a person or organization?
An agent commonly combines several instruction sources. A system policy and user request may be mixed with an email, a document, a website, a calendar entry, an API result, a database record, or a message from another agent. If any of those untrusted sources contains an instruction, the agent may mistake data for authority. NIST’s Center for AI Standards and Innovation (CAISI) calls this agent hijacking, a form of indirect prompt injection in which malicious instructions are inserted into data an agent ingests.
#1 Best Overall
Red teaming therefore evaluates the deployed workflow: the model, orchestration code, retrieval layer, memory, identity, tools, approval process, downstream authorization, logging, and rate limits. A model that refuses a malicious request in isolation can still be dangerous if a poisoned webpage causes it to call an over-privileged tool.
| Part of the workflow | What a red team probes | Possible consequence |
|---|---|---|
| Instructions and data | Can hostile content override the user’s goal or system policy? | Goal hijacking, unauthorized disclosure, misleading output |
| Planner and memory | Will the agent preserve, repeat, or prioritize attacker-controlled content? | Persistent poisoning that affects later sessions or users |
| Tools and identity | Can the agent call tools outside the task or use excessive privileges? | Data changes, administrative actions, financial loss |
| Approvals and downstream systems | Can a persuasive explanation bypass a human or authorization check? | Harmful action approved or executed without independent validation |
| Operations | Can retries, loops, or tool calls run without a practical limit? | Unbounded API, compute, or transaction costs |
Which risks does red teaming find?
Goal hijacking and prompt injection
The attacker’s instruction can be explicit—such as text in an email telling the agent to forward a mailbox—or hidden in a document, image caption, webpage, or API field. The test succeeds when the agent abandons the legitimate task and follows the injected objective. NIST’s evaluations treat this as a workflow failure, not merely a poor-quality answer.
Tool misuse and privilege abuse
An agent may have a legitimate tool but use it in an unsafe way: querying records unrelated to the task, changing a permission, sending a message to the wrong recipient, or invoking an administrative operation. OWASP identifies excessive functionality, excessive permissions, and excessive autonomy as common roots of “excessive agency.”
Free tools Windows power users keep installed
One-click scans. No signup required.
Sensitive-data exposure and exfiltration
Tests look for secrets in prompts, retrieved context, memory, logs, connected storage, and tool responses. An attacker may persuade the agent to place confidential material in an email, a public ticket, a generated file, or a tool argument. CAISI added database-exfiltration tasks to its extended AgentDojo evaluation to make this impact measurable.
Memory and context poisoning
Some agents retain notes, summaries, preferences, or retrieved documents between tasks. A malicious instruction that is stored as a trusted-looking memory can influence future sessions, other users, or another agent. Red teams check whether content is authenticated, scoped to the correct user, reviewable, and removable.
Unexpected code execution
An agent that can run shell commands, notebooks, scripts, or automation may be induced to execute attacker-controlled code. CAISI added remote-code-execution tasks to its extended evaluation. The test should distinguish harmless text describing a command from an actual execution path and record the identity, network access, files, and persistence available to the process.
Supply-chain and inter-agent weaknesses
Third-party tools, plugins, APIs, data feeds, and model-generated messages can be compromised or spoofed. In a multi-agent system, one agent may send a convincing but malicious instruction to another. Testing must validate message origin, intended recipient, allowed actions, and whether trust is being inherited merely because a message came from an agent.
Recommended Free Tools
Cascading failures and approval manipulation
A small error can propagate when several agents share state or automatically trigger one another. A red team also tests the human interface: an agent may present a confident explanation, hide uncertainty, or frame a harmful action as urgent, causing an operator to approve it. Approval is not an effective control if the reviewer cannot see the exact data, tool, parameters, and expected consequence.
Excessive autonomy and high-impact actions
Deletion, payments, account changes, public posts, legal notices, and administrative changes require stronger controls than drafting or searching. Tests ask whether the agent can complete such actions without an independent authorization check, a narrowly scoped approval, or a reversible transaction.
Unbounded consumption
Loops, repeated retries, recursive agent calls, and large tool outputs can create uncontrolled API or compute use. A security exercise should measure whether budgets, timeouts, concurrency limits, and circuit breakers stop the behavior before it becomes an operational or financial incident.
Rank #3
How a realistic agent red-team evaluation works
NIST CAISI used AgentDojo, an open-source framework with simulated Workspace, Travel, Slack, and Banking environments. Each environment supplies tools for legitimate tasks. A scenario gives the agent that task while exposing it to data containing a malicious injection. The evaluation counts a hijack when the agent completes the malicious task.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
CAISI’s enhanced scenarios include remote code execution, database exfiltration, and automated phishing. These additions matter because an aggregate “attack success” number can hide whether the successful action was a trivial unwanted reply or a serious compromise.
- Map the deployed system. Record every model, prompt layer, retrieval source, memory store, tool, credential, network path, human approval, downstream service, and inter-agent channel involved in the target task.
- Define the legitimate objective and prohibited outcomes. Specify what the agent is supposed to do, which data it may access, which tools it may call, and which actions require approval. Write impact definitions before running attacks.
- Seed realistic untrusted content. Place injections in the same emails, files, webpages, API fields, or messages the production workflow reads. Include obfuscated, indirect, and contextually plausible instructions rather than only obvious attack strings.
- Use both baseline and target-adapted attacks. Known attack patterns provide a reference point. Attacks adapted to the target’s tools, prompts, and data often reveal weaknesses that generic probes miss.
- Run repeated attempts when an attacker can retry. Record every run, not only the first success. Probabilistic agents can fail an attack once and comply later.
- Capture the complete trace. Preserve inputs, retrieved passages, memory writes, tool names and parameters, credentials or scopes, approvals, downstream responses, and final side effects. A text transcript alone cannot prove what happened.
- Verify the impact independently. Check whether data actually left its permitted boundary, a command executed, a record changed, a message was sent, or a cost limit was exceeded. Do not classify a stated intention as a completed compromise.
- Retest after controls change. Repeat the same scenario after reducing permissions, adding an approval gate, or changing authorization. Keep attack and control versions so improvements are measurable.
What the published numbers mean
CAISI reported two results in its NIST technical blog, published January 17, 2025 and updated December 19, 2025. They describe particular experiments, not a universal compromise rate for AI agents.
| Result | Experimental context | Correct interpretation |
|---|---|---|
| 11% to 81% attack success | On a held-out Workspace evaluation, CAISI’s strongest novel red-team attack reached 81%, compared with 11% for the strongest baseline attack, using the upgraded Claude 3.5 Sonnet setup described by NIST. | Target-adapted attacks exposed more hijacks in that setup; the range is not a population average. |
| 57% to 80% average attack success | Across five selected injection tasks, the average rose from 57% for one attempt to 80% when each attack was tried 25 times. | Retry opportunities materially change measured risk in that test suite; one-shot testing can understate exposure. |
Task-level impact is more important than a single average. A low success rate on an action that deletes records, transfers money, exposes regulated data, or runs code can justify immediate mitigation. Report the number of attempts, the exact task, the consequence, and whether the result required a novel or known attack.
How to compare red-team results
When comparing two systems, benchmarks, or releases, align the test conditions first. A model-only prompt test cannot be compared directly with an end-to-end workflow that has tools, memory, identities, and external content.
| Comparison axis | Questions to ask |
|---|---|
| Coverage | Does the evaluation include tools, external data, memory, credentials, downstream systems, and multi-agent communication? |
| Attack adaptation | Were attacks generic, previously known, or developed against this specific workflow? |
| Task and consequence | What happened on each task, and how severe would a successful action be? |
| Attempt model | Was each attack tried once, or as many times as a real attacker could retry? |
| Controls | Were least privilege, approval gates, authorization checks, monitoring, and rate limits active and tested? |
| Evidence quality | Are tool traces and side effects available, or is the score based only on generated text? |
NIST cautions that evaluations require continual improvement as systems and attacks change. A benchmark score is evidence about the tested configuration, not a safety guarantee for a different prompt, tool set, model version, or production identity.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Defenses that address the findings
Give the agent the minimum authority
Expose only the functions needed for the task. Separate read and write tools, remove unused endpoints, restrict records by tenant and purpose, and use short-lived credentials with narrow scopes. A system that cannot delete, export, or administer data cannot be hijacked into those actions through prompt injection.
Enforce authorization outside the model
Every downstream request should be checked against a security policy using the authenticated user, resource, action, and context. Do not ask the model to decide whether its own proposed action is permitted. The service receiving the request must reject unauthorized parameters even when the agent is manipulated.
Require meaningful approval for high-impact operations
Use approval for payments, deletion, privilege changes, external publication, code execution, and other irreversible or high-consequence actions. Show the exact operation, destination, data, and consequence to the reviewer. Make approvals single-use and bind them to the specific parameters so an agent cannot substitute a different action after approval.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Treat external content as untrusted
Keep instructions and data clearly separated in prompts and intermediate representations. Label provenance, validate and sanitize inputs, constrain retrieved content, and prevent documents or webpages from directly issuing tool commands. Sanitization reduces some attack surface but cannot replace authorization and least privilege.
Best Value
Protect memory and agent-to-agent messages
Apply access control, provenance, expiry, and review to stored memories. Do not let one user’s content become a global instruction. Authenticate inter-agent messages, define allowed message types, and validate each requested action independently rather than inheriting trust from the sender.
Limit retries, resources, and observability gaps
Set time, token, tool-call, recursion, concurrency, and spending limits. Add circuit breakers for repeated failures or suspicious destinations. Log enough context to reconstruct decisions while protecting secrets, and alert on unusual exports, privilege requests, repeated denials, or tool-call bursts.
Using open-source evaluation resources
AgentDojo provides simulated tasks and environments for studying agent hijacking, while CAISI’s open-source enhancements add higher-impact scenarios such as code execution and exfiltration. These resources are useful for repeatable regression tests, but a simulation cannot represent every production dependency or permission.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteNVIDIA’s garak project describes a free, open-source vulnerability scanner for probing language models or broader systems. A scanner can broaden automated coverage; it should complement, not replace, task-specific scenarios, trace review, and verification of real side effects.
What a useful report contains
- Scope: model and version, tools, identities, data sources, memory, environment, and dates.
- Attack description: injection location, attacker capability, adaptation method, and number of attempts.
- Observed path: retrieved content, model decision, tool calls, approvals, downstream authorization, and side effects.
- Impact: data exposed, records changed, code run, message sent, privilege gained, or resources consumed.
- Control assessment: which boundary stopped the attack, which failed, and whether the failure was deterministic or probabilistic.
- Remediation and retest: permission changes, approval requirements, policy enforcement, monitoring, and the results of a repeat evaluation.
The practical objective is not to make an agent appear harmless in a demo. It is to ensure that hostile content cannot convert the agent’s capabilities and authority into an unauthorized real-world action.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
