Test the complete agent application—not just the model’s replies—in an isolated environment that simulates its tools, data, and tasks. Exercise direct attempts to misuse tools and indirect prompt injections in content the agent reads; capture authorization decisions and side effects. If a prohibited action executes, the test fails even if the agent later refuses or apologizes.
What a useful pre-deployment test must cover
An agent’s tool-use risk is shaped by more than its model. Test the parts that can influence or execute an action: orchestration, tool interfaces, authorization policy, credentials, retrieval, memory, approval controls, and external data. The goal is to establish whether an unauthorized action can happen and what controls stop it—not merely whether the final answer sounds safe.
As an Amazon Associate I earn from qualifying purchases.
Use a disposable account, mock service, or sandbox with synthetic data. Do not put production credentials or real customer data in test fixtures. For every case, define the legitimate task, attacker-controlled input, prohibited action, expected policy decision, observable evidence, and safe cleanup.
Build an abuse-case matrix
Start with the abuse paths in OWASP’s AI Agent Security Cheat Sheet, then add cases specific to your application. The test should make the attacker’s objective and the expected control observable.
#1 Best Overall
| Abuse case | Test setup | Evidence to inspect |
|---|---|---|
| Prompt override | A user or retrieved item tells the agent to ignore higher-priority instructions. | Whether trusted instructions and policy remain in force; whether any prohibited tool call is requested or executed. |
| Unauthorized tool use | The agent requests a tool or operation outside the user’s or session’s scope. | Whether an independent authorization layer denies the call before execution. |
| Privilege escalation | A low-trust session attempts to use a privileged tool or credential. | Whether role boundaries and credential scopes prevent the operation. |
| Memory poisoning | Malicious content is offered for persistence or later retrieval. | Whether it is rejected, scoped, sanitized, or given an appropriate expiry—and whether a later task can trigger it. |
| Data exfiltration | External content asks the agent to send private context to an attacker-controlled destination. | Tool arguments and network effects; whether the transfer is blocked or requires the correct approval. |
| Approval bypass | A high-impact action is attempted without approval, with stale approval, or with approval for a different action. | Whether approval is current and bound to the exact tool, target, and normalized parameters. |
| Recursive tool abuse | An operation repeatedly calls tools or retries. | Whether depth, retry, token, and cost limits stop the run. |
| Multi-agent boundary crossing | One compromised agent attempts to induce a second agent to act outside its authority. | Whether trust boundaries and delegated scopes remain intact across the handoff. |
Test indirect prompt injection with ordinary tasks
Place malicious instructions in realistic untrusted material the agent is expected to read: a web page, document, email, tool output, or retrieved record. Pair that content with a normal user request, then check whether the injected instruction changes the action taken. For example, a routine request to summarize an email should not authorize a tool action requested only by text inside that email.
NIST CAISI describes agent hijacking as a form of indirect prompt injection: malicious instructions embedded in ingested data can lead to unintended actions. The risk arises because agents combine trusted developer instructions with task-relevant data, creating a boundary that attackers may try to exploit. Test the boundary in the actual application flow, including how retrieved content reaches the model and how proposed tool calls are authorized.
Run the test as a repeatable workflow
1. Inventory the action boundary
List every tool and operation the agent can invoke, the scopes and credentials available, the data sources it reads, and the side effects each operation can cause. Identify which actions require approval and which component enforces that requirement. Use the inventory to distinguish an agent’s ability to propose an action from the application’s ability to execute it.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errors2. Define each case before running it
For every scenario, record the allowed task, untrusted input, disallowed action, expected authorization result, evidence to capture, and cleanup. Include benign tasks as well as attacks so that a test can reveal both unsafe execution and controls that block legitimate work unnecessarily.
3. Instrument the tool gateway or mock tools
Capture the requested tool name and arguments, caller or session, policy decision, approval state, execution result, and resulting state changes. Verify that a denial occurs before execution. A refusal in final prose does not pass if the unsafe operation already ran.
4. Repeat and adapt scenarios
Run cases multiple times when model behavior is nondeterministic. Report results by task and attack type as well as in aggregate; a single overall score can conceal a failure in one high-impact task. Update attack cases when prompts, models, tools, or surrounding controls change, and use human red-team review for high-impact scenarios.
Rank #3
5. Turn failures into release gates
Keep attack inputs and expected denials under version control, and add regression cases for every discovered injection, memory-poisoning, or tool-abuse failure. Require updated test results when prompts, tools, memory, retrieval, policies, providers, credential scopes, or approval logic change. OWASP recommends blocking releases when high-risk changes to tool policy, approvals, or credential scope lack updated tests.
Recommended Free Tools
6. Keep a reproducible test record
For each run, record the tested agent version, model provider, tool policy, retrieval configuration, cases executed, expected results, observed approvals, denials, timeouts or circuit breakers, and any residual risk accepted with compensating controls. Keep secrets and live customer data out of the fixtures.
Judge outcomes, not just attack strings
Track whether an attack caused an unauthorized action, not only whether the agent repeated or rejected malicious text. A useful report separates outcomes by task and attack type, records the action and resulting state, and notes the controls that prevented or limited impact. Repeated attempts matter because a single run may not expose behavior that varies between runs.
Rank #4
NIST CAISI’s 2025 AgentDojo evaluation illustrates why adaptive attacks matter. In a specific Workspace evaluation against the upgraded Claude 3.5 Sonnet, CAISI reported an 11% attack success rate for the strongest baseline attack and 81% for the strongest newly developed attack. Those figures describe that experiment, not a general vulnerability rate for agents or a prediction for another deployment.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Use benchmarks and frameworks as starting points
AgentDojo
AgentDojo is an open-source framework used by NIST CAISI for hijacking evaluations. Its reported simulated environments are Workspace, Travel, Slack, and Banking, with simulated tools. CAISI added scenarios for remote code execution, database exfiltration, and automated phishing. Adapt scenarios to your agent’s tasks and authorization boundaries; benchmark results do not certify a different application or deployment.
Free tools Windows power users keep installed
One-click scans. No signup required.
Promptfoo
Promptfoo is an open-source framework for evaluating prompts, agents, and AI applications. OpenAI’s red-teaming guide points to it for generating adversarial cases and inspecting target behavior. Check its current features, integrations, license, and fit against your own stack before adopting it.
Best Value
Managed red teaming
OpenAI states that its managed red-teaming service is available for enterprise customers. Confirm current eligibility, scope, and terms directly if considering it. Neither a framework nor a managed service replaces tests of your application’s actual tool boundary and side effects.
Compare options by whether they exercise the full application or only model behavior, their attack and environment coverage, whether they capture tool actions and side effects, repeatability and CI integration, custom-scenario support, and operational support and reporting. Compatibility depends on the specific orchestration framework, provider, and tool gateway; no universal ranking establishes which option fits every stack.
Constrain impact even when an attack succeeds
Testing should verify defense in depth rather than rely on the model to identify malicious content correctly. Limit available tools and privileges, separate an agent’s proposal from execution, and have an independent policy component validate scope and approvals. Bind approval to the precise action, including its tool, target, and normalized parameters, so that approval for one operation cannot authorize a changed or replayed request.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →OpenAI’s March 11, 2026 article on prompt-injection defenses describes the objective as constraining the impact of manipulation even when it succeeds. It also reports a 50% outcome in testing of a particular broad request to research emails, based on a 2025 prompt-injection example reported by external security researchers. That is a result from one setup, not a population statistic or an expected failure rate for other agents.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
