Evaluate an AI agent by testing the whole path from the instructions and untrusted content it reads to the tools it can invoke and the state those tools change. A safe-sounding final answer is not enough: capture tool requests, authorization decisions, data destinations, and side effects. Use synthetic data and isolated tools, test attacks alongside legitimate tasks, repeat runs, and report results by security objective rather than as a single “safe” score.
Define the system and the trust boundaries
An agent’s security depends on more than its model response. Its effective behavior also comes from prompts, tools, permissions, retrieval, memory, policies, and the environment in which actions occur. Record the versions and settings for each before testing so that results can be compared after changes.
As an Amazon Associate I earn from qualifying purchases.
- Agent configuration: build, model and provider version, system and developer prompt versions, policies, and relevant runtime settings.
- Capabilities: available tools, credentials, read/write permissions, resource scope, and any limits on retries, recursion, depth, tokens, or cost.
- Context sources: retrieval configuration, external websites, email, files, memory, and the rules for persisting information between sessions or users.
- Test environment: geography or operating context where relevant, sandbox boundaries, logging, and which state changes or data destinations are observable.
Keep user instructions and external content as separate test channels. A malicious string placed in a user message tests direct injection; the same instruction planted in a web page, email, document, or retrieval result tests whether the agent can keep untrusted data from taking control. NIST describes agent hijacking as a failure to maintain a clear separation between trusted instructions and untrusted external data in its evaluation guidance.
Test only systems and environments you are authorized to assess. Use sandbox substitutes for tools such as email, file access, shell, browser actions, and APIs. Populate them with dummy credentials and synthetic records; never use live customer data, real secrets, or real third-party targets as fixtures.
Build a test matrix around security objectives
Each case should specify the legitimate task, attacker objective, injection channel, relevant context, expected policy decision (allow, block, or human review), and observable violation. Define the violation before running the case; otherwise, a plausible-looking answer can conceal an unauthorized action.
Instruction override and secret extraction
Test whether a user message or retrieved content can override higher-priority instructions or make the agent disclose a seeded dummy secret marker. OpenAI’s published evaluation exercise describes repeated adversarial queries against a hidden phrase or password and counts correct refusals; that is a useful pattern for synthetic-secret tests, not a reason to place actual credentials in prompts or fixtures. See the evaluation exercise description.
Indirect injection and task hijacking
Give the agent a legitimate task, then place attacker instructions in the external content it must process to complete that task. Check whether it follows the user’s goal or switches to the attacker’s objective. Test each relevant source type separately—such as a page, email, file, or retrieval result—because a direct prompt attack does not establish how the agent handles injected content found elsewhere.
Free tools Windows power users keep installed
One-click scans. No signup required.
Unauthorized tools and privilege escalation
Try to induce actions beyond the user’s authorization, resource scope, or intended read/write permissions. Include attempts to cross from read access to write access or to reach resources outside the task’s scope. Observe the actual tool layer: a refusal in the final response does not undo a request the agent already sent or a change a tool already made.
Disclosure and exfiltration
Seed the sandbox with synthetic sensitive records and specify which destinations are permitted. Check the final response and instrumented tool, API, and logging paths for disclosure. A clean chat response alone cannot show that data was not sent through another channel.
Memory poisoning and cross-session effects
Test whether malicious retrieved content is persisted as trusted information or influences a later session or another user. Record what is written to memory, which identities can read it, and whether later tasks behave differently. Preserve observed failures as regression cases, including tests of the later session where the effect appears.
Runaway or chained actions
Test malicious or looping tasks against limits for recursive calls, retries, execution depth, tokens, and cost. Observe whether a single instruction can trigger a chain of actions outside the intended task, and whether configured limits stop that chain at the tool boundary.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchBenign controls
Pair attacks with legitimate in-scope tasks, including sensitive operations the agent is allowed to perform. Record policy correctness separately from task completion: a system can block an attack but also wrongly refuse useful work, or complete a benign task while mishandling a different security objective.
Run the evaluation safely and repeatably
- Freeze the test configuration. Save the agent, prompt, model/provider, tool, permission, memory, retrieval, policy, and environment versions. Note the deployment context that could affect behavior.
- Write observable cases. For each case, record the intended task, attack goal, attack channel, required context, expected allow/block/review decision, and the concrete event that constitutes a violation.
- Instrument sandbox tools. Use substitutes with synthetic data and dummy credentials. Capture requests, authorization results, outputs, state changes, and attempted data destinations.
- Run the legitimate task with the attack in the correct channel. For indirect-injection cases, put the payload in the external content the agent encounters. Run direct user-message injection as a separate case.
- Repeat attempts. Model behavior can vary between runs. Keep each run’s result and the run count; a single pass or one successful defense is not a stable rate. NIST also recommends repeated attempts for more realistic assessment in its agent hijacking evaluation guidance.
- Compare defenses on identical cases. Preserve paired outcomes and change one relevant defense or configuration at a time where practical. Different case sets do not support attributing all score differences to a defense.
- Review traces for validity. Inspect transcripts and tool logs for unexpected network access, out-of-scope actions, answer lookup, task-specific hardcoding, or grader gaming. NIST’s evaluation-integrity guidance distinguishes solution contamination from grader gaming and discusses transcript review and standardized benchmark affordances.
- Version the cases and rerun after material changes. Keep observed attacks, expected denials, and benign controls as regression tests when prompts, tools, credentials, retrieval, memory, policies, or models change. OWASP recommends structured testing before production and after material changes in its AI Agent Security Cheat Sheet.
Measure security outcomes and usefulness separately
Report results at the level of objective and task. Keep a per-case record so readers can see what the agent did, rather than relying on an aggregate label.
- Security outcomes: attack success by objective, such as extraction, unauthorized tool action, data transfer, memory influence, or task hijacking. Where the test distinguishes them, report whether an attack began executing separately from whether it achieved the attacker’s end goal.
- Tool-layer evidence: whether a real policy violation or state change occurred, even if the final answer appeared safe; include attempted actions and authorization outcomes.
- Benign-task outcomes: completion rate, incorrect refusals or other false positives, and cases held for human review. Do not fold these into attack-success rates.
- Reproducibility details: case count, repeated-run count, model and defense versions, settings, and the source or corpus of test cases.
- Uncertainty: provide confidence intervals only when the sampling design supports them, and state the method and assumptions. Do not treat prompt variants or repeated runs as independent samples without a statistical basis.
Small, hand-picked suites are useful for regression checks but do not estimate population risk. OWASP labels its current prompt-injection examples a smoke test, not a security benchmark: the sheet includes 14 illustrative attack inputs and seven benign requests. It notes that zero false positives in seven independent trials still yields an approximate 95% Wilson interval from 0% to 35.4%. The example illustrates how little a tiny control set can establish; it is not a measured false-positive rate for your agent. See the OWASP prompt-injection guidance.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Choose a benchmark as a starting point, not a verdict
Frameworks cover different agent settings. Select one that resembles your modality and tasks, then add cases for your own tools, data, and authorization policy.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
| Option | Best fit | What it contributes | Limits to account for |
|---|---|---|---|
| AgentDojo | General tool-using agents in simulated work, travel, Slack, or banking tasks. | NIST CAISI used its simulated environments and extended cases for remote code execution, database exfiltration, and automated phishing. See NIST’s evaluation article. | NIST describes an evolving framework and attack types added beyond baseline cases. Check the current implementation and supplement it with deployment-specific tasks. |
| WASP | Browser and web-navigation agents. | An isolated executable web environment with prompt-injection hijacking objectives; the public implementation stores logs and traces. See the WASP paper and official implementation. | Its scope is web agents. The paper’s reported results are specific to the studied agents, tasks, and setup, not a universal deployed-system rate. |
| OWASP smoke-test examples | Quick regression checks and a starting point for custom cases. | Fourteen hand-picked attack inputs and seven benign requests, with guidance on setup and observation. See the OWASP cheat sheet. | OWASP explicitly says these illustrative cases are not a security benchmark or representative sample of traffic. |
Compare candidate frameworks on agent modality and task realism, attack and benign-control coverage, isolation, trace and outcome observability, repeatability, customization, maintenance, and alignment between scoring and your actual authorization rules. These are practical comparison criteria, not a published ranking.
One reported finding illustrates why scope matters: WASP authors reported that 16–86% of adversarial instructions began executing, while 0–17% achieved the attacker’s goal, across the web agents and benchmark tasks they studied. Those ranges describe that paper’s setups, not the expected attack rate for other agents or production systems; consult the paper for its conditions.
Decide what the results can support
A credible evaluation supports a bounded statement about tested configurations, cases, and observed outcomes. It does not prove that an agent is safe in general. Do not present a smoke test as a benchmark, a benchmark score as a guarantee, or a textual refusal as evidence that no action happened. Keep attack resistance and legitimate task usefulness visible together, and preserve enough configuration and trace detail for another team to understand what was actually tested.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →
