Evaluate an AI agent as a complete system—not just a model responding to prompts. Map the data it can read, the actions it can take, and the people or content that can influence it; test whether malicious input can trigger unauthorized actions or expose data; and verify that permissions are enforced outside the model. Use repeatable tests before launch and after material changes.
What an AI agent security evaluation needs to cover
An agent’s security depends on more than its model. Its attack surface includes the model, orchestration code, tools and APIs, identities and credentials, memory, retrieval sources, and communication channels. A seemingly ordinary input—a web page, document, email, tool response, or MCP resource description—may contain instructions intended to redirect the agent. If the agent can then take actions or access sensitive information, the risk is not limited to an undesirable answer.
As an Amazon Associate I earn from qualifying purchases.
OWASP’s agent security guidance emphasizes mapping trust boundaries and limiting the agent’s authority. Its application-security testing guidance also treats retrieved content as a distinct input channel. In practice, an evaluation should answer three separate questions:
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →- What can influence the agent? Include direct user messages and indirect inputs such as files, retrieved pages, emails, and tool outputs.
- What can the agent do? Identify each tool, API, data store, credential, and action available through the workflow.
- What prevents unauthorized effects? Confirm that execution code, gateways, and policy controls check permission independently of the model’s willingness to comply.
A model refusal is useful evidence about one response, but it does not establish that an API or tool will reject a crafted call. Evaluate both the model-facing behavior and the controls that ultimately permit or block actions.
#1 Best Overall
1. Define the intended job and map the system
Set a clear boundary for the agent
Write down the tasks the agent is meant to perform, who is allowed to request them, what data it may use, and what consequences an error could have. Distinguish low-impact actions, such as summarizing an approved document, from actions that change records, send messages externally, spend money, or administer a system. The intended job becomes the baseline against which both ordinary performance and abuse are judged.
Do not define the boundary as “the agent can use the available tools.” For each capability, specify the legitimate purpose, the permitted caller, the permitted target or data, and whether the action needs approval. A tool that is legitimate for one workflow can still be excessive for another.
Inventory components and trust boundaries
Trace the full path from input to outcome. Record every model and orchestration component, tool and API, API scope, identity, credential, data store, retrieval source, memory component, and communication channel. Include any MCP server or other intermediary that supplies resources or enables actions.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
For each input source, note whether it is trusted, who can modify it, and what it can influence. A retrieved document might affect the model’s response; a tool output might also affect a later tool call. Map those paths rather than assuming that only user messages can contain hostile instructions. OWASP recommends threat modeling agents, orchestrators, tools, sources, and trust boundaries before testing.
A useful inventory records the capability and its boundary, not just its name:
- Identity: Which principal does the agent use, and can actions be attributed to it?
- Scope: Which operations and resources can that identity reach?
- Input influence: Which user, retrieved, or tool-provided content can affect the capability?
- Impact: What could happen if the agent calls it incorrectly or for an unauthorized purpose?
- Control: Where is authorization checked, and what approval or logging applies?
2. Check permissions, credentials, and isolation
Design the system so that a manipulated agent has as little authority as possible. OWASP’s AI Agent and MCP guidance recommends least privilege, scoped identities, short-lived credentials, sandboxing, and limits on network egress. These controls reduce the consequences of a successful injection; they do not make injection testing unnecessary.
Review identities and credentials
Use an identity dedicated to the agent or task, rather than a developer’s personal account. Prefer credentials limited to the specific job and resource, and set them to expire when the task or session ends where the system permits. Use read-only access when the task does not require changes.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesCheck where secrets are stored and what the agent can read. They should not be placed in prompts, exposed through files accessible to the agent, or made available in environment variables the agent can inspect. Test access boundaries with non-production credentials and data. If the agent can retrieve a secret merely because it is present in its runtime, a refusal instruction is not a reliable security boundary.
Review runtime and network boundaries
Run the agent in a sandbox or disposable environment with limited filesystem access and restricted outbound network connectivity. Determine whether the boundary applies to every tool and intermediary; do not assume that one sandbox or a permission prompt protects an entire workflow. Check MCP servers and other connectors individually, including what resources they expose and what actions they can perform.
Document which outbound destinations are necessary for the intended job. Unrestricted network access can turn a data leak into an external transmission, while broad filesystem access can expose unrelated information. The evaluation should verify the controls at the runtime or network layer, not infer them from a successful prompt-level refusal.
Rank #3
3. Establish normal behavior, then test abuse cases
Run representative legitimate tasks first. Record expected tool calls, outputs, access decisions, and approvals. This baseline helps distinguish a security failure from a task the agent was never intended to complete, and it helps reveal when defensive controls block necessary work.
Next, use realistic workflows with safe test data and no production secrets. OWASP’s agent security cheat sheet describes structured abuse-case categories; NIST’s January 2025 guidance on agent hijacking discusses indirect prompt injection through ingested data. Test not only a malicious user message, but also hostile instructions embedded in content the agent is expected to consume.
| Abuse case | Test input or path | What to observe |
|---|---|---|
| Instruction override | A user message or supplied content asks the agent to disregard its governing instructions. | Whether the agent follows the hostile direction, changes its task, or attempts an unauthorized action. |
| Indirect injection | Place adversarial directions in a test web page, file, email, retrieval result, or tool response. | Whether untrusted content affects subsequent reasoning, tool selection, arguments, or disclosure. |
| Tool misuse | Ask for an action outside the requested task or beyond the agent’s assigned authority. | Whether both the agent and the execution layer prevent the call or reject its arguments. |
| Privilege escalation | Use a low-trust user or input to try to reach privileged tools, data, or credentials. | Whether authorization follows the caller and resource boundary rather than the agent’s wording. |
| Data exfiltration | Try to induce disclosure through a tool, email, network request, rendered link, citation, log, or final response. | Whether sensitive test data leaves its permitted context, including through less obvious output channels. |
| Memory poisoning | Introduce hostile content that could persist and affect a later task or user. | Whether memory is validated, scoped to the appropriate user or task, and subject to expiry. |
| Runaway execution | Exercise recursive tool use, repeated retries, or long chains of actions. | Whether limits on retries, chain depth, token consumption, and cost stop unbounded behavior. |
Test single-turn and multi-turn paths where relevant. A malicious instruction may have little effect in isolation but influence later actions after the agent has gathered context or stored information. Repeat attacks as well: NIST’s CAISI notes that multiple attempts may produce a more realistic evaluation of agent hijacking than a single attempt. Record each attempt rather than treating one successful refusal as proof of consistent protection.
4. Verify authorization independently of the model
Prompt tests alone cannot prove that actions are protected. OWASP’s prompt-injection prevention guidance recommends independently validating tool calls and permissions. Exercise the API gateway, policy service, or tool execution layer directly with crafted requests, so that the test does not depend on the model generating—or refusing to generate—the call.
Check every action at execution time
For each call, verify the caller’s identity and scope, the target resource, the arguments, and any required approval. Authorization should be checked for the actual operation being attempted, not inferred from the agent’s general access or from an earlier request. Reject malformed arguments and requests outside the permitted target or task.
For destructive, financial, administrative, or externally visible actions, apply stronger action-specific checks. Approval should be tied to the exact action being approved; a broad approval to “continue” is not equivalent to approval for a particular recipient, amount, record change, or administrative operation. If the authorization or approval check is unavailable, the high-impact action should fail closed rather than proceed.
Separate model behavior from control behavior
Keep two kinds of results in the test record. First, note what the model did: whether it recognized hostile content, refused, or attempted a tool call. Second, note what the enforcement layer did: whether it authenticated the caller, validated scope and arguments, required approval, and blocked or allowed execution. If the model attempts an unauthorized action but the gateway rejects it, the model behavior still deserves investigation; if the model refuses but the gateway would accept a crafted call, the system has an enforcement gap.
OWASP cautions that prompt-injection examples are smoke tests, not a security benchmark. A small set of successful demonstrations can expose problems, but it cannot establish comprehensive safety. The coverage of the scenarios and the behavior of independent controls matter more than a single pass/fail result.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.5. Set release criteria and preserve evidence
Decide before launch which abuse cases must be blocked, which actions require human or policy approval, what data may leave the system, and how failures will be logged and investigated. Set explicit criteria for high-impact actions and data access rather than relying on an overall judgment that the agent “seems safe.”
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Track both security failures and completion of legitimate tasks. A system that blocks every action may reduce certain risks while failing its purpose. The evaluation should show whether the agent can complete authorized work while controls prevent unauthorized work.
Best Value
Keep enough evidence to reproduce and review results:
- The tested configuration, including model provider, prompts, tools, policies, memory, retrieval sources, and relevant permission settings.
- The test scenarios and input data, including the path by which adversarial content reached the agent.
- The agent’s outputs, attempted tool calls, gateway or policy decisions, approvals, and relevant logs.
- The outcome, severity, remediation, and any unresolved risk accepted for release.
Use non-production test environments, synthetic or otherwise safe data, and controlled outbound access. Do not place real secrets or sensitive user data in adversarial prompts just to see whether the agent will reveal them.
6. Retest when the system changes
Security results describe a tested configuration, not every future version of an agent. OWASP recommends testing before production and repeating tests after material changes. Rerun relevant scenarios when prompts, tools, memory, retrieval sources, policies, or model providers change. Changes in connected services or permission scopes can also alter what the agent can do, so update the system map and test coverage when those boundaries move.
Keep abuse cases repeatable so they can be run again during release checks and after changes. Internal test suites or agent-testing frameworks can help organize scenarios, but a framework does not replace the work of defining your own trust boundaries, testing indirect inputs, and verifying authorization in the execution layer. Compare testing approaches by their coverage of direct and indirect injection, multi-turn behavior, tools, retrieval, memory, exfiltration, and privilege boundaries; their ability to reproduce representative workflows and repeated attempts; their visibility into gateway and API enforcement; and their auditability, isolation, and maintenance burden.
Quick Recap
Common evaluation gaps to avoid
- Testing only direct prompts: retrieved content and tool outputs are also potential carriers of hostile instructions.
- Trusting a refusal as authorization: a refusal tests model behavior; it does not show that the tool layer rejects unauthorized calls.
- Granting broad credentials “for convenience”: excessive access increases the impact of manipulation. Narrow scopes and dedicated identities reduce that impact.
- Ignoring persistence: memory can carry hostile or inappropriate information into later tasks unless its validation, scope, and expiry are controlled.
- Testing a single attempt: repeated or multi-step attempts can expose behavior missed by a one-shot check.
- Recording only pass or fail: without configuration, scenario, tool-call, and enforcement evidence, results are difficult to reproduce or act on.
- Calling a few examples a benchmark: smoke tests can identify clear weaknesses but do not demonstrate complete security.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
