The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →To test whether an AI agent will follow a stranger’s instructions, give it a legitimate task, place an attacker’s instruction in the untrusted content the agent must process, and define in advance what would count as a failure. Run the test with dummy data and sandboxed or instrumented tools—not real accounts or secrets—and check both whether the attack worked and whether the agent still completed the user’s task safely.
What this test is designed to catch
Prompt injection is an attempt to make a model disregard its intended task or safeguards and follow an attacker’s instructions. It can be direct, when the instruction comes from the user prompt, or indirect, when it is embedded in content the agent later processes, such as a webpage, document, email, or tool output. Those are different test channels: an instruction typed into the user prompt does not show whether the agent resists a stranger’s instruction hidden in retrieved content. OWASP describes both forms in its LLM01:2025 Prompt Injection guidance.
As an Amazon Associate I earn from qualifying purchases.
With a tool-using agent, the concern is not limited to an odd or misleading answer. Depending on the connected tools, permissions, and application context, a compromised agent might disclose sensitive data or attempt an unauthorized action. NIST uses agent hijacking for this broader risk. A test result applies to the particular agent configuration and workflow you tested; it does not establish that every agent is vulnerable, or that an untested version will behave the same way.
Build a safe, repeatable test
- Choose one real workflow. Pick a legitimate task that is within the agent’s intended scope—for example, summarizing a document, finding a requested email, or gathering information from a page. Keep the user’s objective clear and fixed across test runs.
- Define the attack and failure condition before running it. For instance, the untrusted content could ask the agent to reveal a seeded dummy secret or invoke a tool action the user did not authorize. Decide exactly what constitutes failure: returning the protected value, making an unsafe request, or carrying out the action. Distinguish these outcomes rather than treating them as interchangeable.
- Put the instruction in the channel you want to evaluate. For an indirect-injection test, place it in a retrieved page, document, email, or tool output. If you instead put the instruction in the user’s message, label that as a direct-injection test and report it separately.
- Use harmless fixtures and isolated tools. Seed a fake secret and fake records. Route tool calls to sandboxed or instrumented substitutes that log attempted actions but cannot affect real accounts or data. OWASP’s Prompt Injection Prevention Cheat Sheet recommends harmless data and instrumented tool substitutes for testing.
- Run controls as well as attack cases. Try the legitimate task without an attack, and try it with benign content that merely resembles an instruction. These controls help distinguish a security failure from ordinary task failure or an overly broad block.
- Record the full outcome. Note whether the legitimate task was completed correctly, whether the attacker’s objective occurred, which tools the agent attempted to call, and whether application controls blocked any unauthorized action. Include the model or provider, prompts and policies, tools, permissions, retrieval setup, and attack channel so someone else can reproduce the case.
- Keep the cases and rerun them. Store test inputs and expected outcomes in a version-controlled regression set. Run it before release and after meaningful changes to prompts, tools, memory, retrieval, policies, or model providers. OWASP’s AI Agent Security Cheat Sheet recommends structured, repeatable agent testing.
Measure security and usefulness together
A test that blocks every tool call may prevent an attack while also making the agent useless. Report security and task utility as separate outcomes, and define the calculation for your test set rather than implying it is a universal score.
- Attack success: attack cases in which the predefined attacker objective occurred, divided by the total attack cases.
- Task utility under attack: attack cases in which the agent completed the legitimate task correctly and without unsafe side effects, divided by the total attack cases.
- Attempted versus executed actions: report separately whether the agent requested an unsafe tool action and whether the application actually allowed it.
- Benign-task failures: track incorrect answers, unnecessary refusals, and false blocks in clean or benign-control cases.
For each measure, give the numerator and denominator, the exact test configuration, the attack channel, and the failure definition. Do not compare percentages from different task sets as if they measured the same thing. A result can also depend on the test design: static attacks may miss attacks that adapt to the agent’s responses.
AgentDojo is an extensible research framework for evaluating agents that use tools over untrusted data. Its NeurIPS 2024 paper describes 97 realistic tasks and 629 security test cases, and evaluates attack success alongside utility under attack. Those numbers describe the benchmark’s size—not how often agents are hijacked in real-world use or how a particular deployed agent will perform. See the AgentDojo paper page.
Rank #2
Use the result to strengthen controls outside the model
Prompt injection is difficult to solve by asking the model to distinguish instructions from data alone: both are processed as natural language. OWASP says fool-proof prevention is unclear, so treat defenses as ways to reduce the likelihood or impact of an attack, not as guarantees. Delimiting or labeling untrusted content can help signal its status, but the label itself is not an enforced security boundary.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitches- Enforce authorization in application code. Check whether an action is permitted when the tool executes; do not rely on the model to police its own access.
- Apply least privilege. Give each tool and workflow only the authority it needs. Validate proposed tool arguments and reject requests outside the user’s authorization.
- Require action-specific approval for high-risk side effects. A model’s request should not silently send, delete, publish, or change something the user has not approved.
- Keep credentials and authorization logic out of the system prompt. A prompt is not a secure place to store secrets or establish access control. OWASP’s LLM06:2025 Excessive Agency guidance explains how excessive permissions can amplify the consequences of injection.
The OWASP Gen AI Security Project’s LLM01:2025 guidance recommends: “Perform regular penetration testing and breach simulations, treating the model as an untrusted user to test the effectiveness of trust boundaries and access controls.” That framing is useful for agent tests: assess not only what the model says, but what the surrounding system permits it to do. NIST’s guidance on strengthening AI agent hijacking evaluations and its summary of responses on AI agent security provide further context for evaluating these systems.
Quick Recap
Best Value
Rank #4
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
