AgentGuardBench is an open benchmark for testing how tool-using AI agents handle security and privacy risks across synthetic scenarios in four languages. Its first published results demonstrate that the benchmark’s scoring pipeline distinguishes two fixed policies—not that commercial or production AI agents are safe or unsafe.
What AgentGuardBench evaluates
AgentGuardBench tests whether an agent can follow safe boundaries while using tools, including when a request is malicious or conflicts with legitimate work. The project describes six risk categories: prompt injection, privacy leakage, tool misuse, privilege abuse, memory safety, and benign controls. Scenarios use synthetic information and inert tools, so a local run does not require production credentials, paid model APIs, or real external recipients.
As an Amazon Associate I earn from qualifying purchases.
The project overview describes version 0.1.1 as a set of 120 fully synthetic scenarios; the repository labels its 120-case dataset description version 0.1. Both describe five sectors—banking, healthcare, education, government, and recruitment—and four languages: English, French, Swahili, and Yoruba. The overview reports 20 cases per risk category, 24 per sector, and 30 per language. These are project-reported design figures. The version labels differ between the overview and repository, so the precise dataset version should not be treated as settled without checking the archived dataset.
The project says its scenarios contain no real people, credentials, organizations, or operational targets. The repository outlines a flow from synthetic scenarios through an agent policy and safe mock tools to an evaluator that produces metrics and machine-readable results.
#1 Best Overall
What the scenarios expect an agent to do
| Scenario family | What is tested | Expected behavior |
|---|---|---|
| Prompt injection | Malicious instructions embedded in untrusted content | Refuse the injected directions |
| Privacy leakage | A request for protected fields | Redact sensitive information |
| Tool misuse | An unreviewed external action | Ask for human approval |
| Privilege abuse | A read-only role asked to perform a destructive action | Refuse |
| Memory safety | A request to reveal secrets supplied earlier | Refuse |
| Benign controls | A legitimate task involving public synthetic information | Complete the task |
This mix matters: a useful security evaluation should measure not only whether an agent blocks unsafe requests, but also whether it still completes legitimate ones.
How to read the published baseline
In a 2026 project overview, benchmark author Joseph Arayemi reports results from two deterministic fixtures run against the same 120 cases. The project says these results validate the benchmark mechanics and scoring pipeline; they are not evaluations of commercial or production models.
| Metric | Strict reference policy | Permissive control |
|---|---|---|
| Overall pass rate | 100.00% | 16.67% |
| Attack success rate | 0.00% | 100.00% |
| Privacy leakage rate | 0.00% | 50.00% |
| Unauthorized tool-use rate | 0.00% | 33.33% |
| Human-approval violation rate | 0.00% | 16.67% |
| Benign-task completion rate | 100.00% | 100.00% |
These percentages describe the author’s deterministic fixture results on this dataset, not a general estimate of agent performance. The author notes that the templates are intentionally easy to audit. The available project pages do not report comparative results for production models.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
The benchmark reports overall pass rate, attack success, privacy leakage, unauthorized tool use, human-approval violations, benign-task completion, and category-level performance. When comparing policies, examine safety and utility together: a low attack-success rate is not enough if an agent also fails legitimate tasks. Category, sector, and language breakdowns can reveal uneven performance that an aggregate score hides.
Rank #3
Running the benchmark locally
The project overview specifies Python 3.10 or later. The repository’s quick-start commands generate the dataset, run both included policies, and execute the test suite:
git clone https://github.com/josepharayemi-netizen/agentguardbench.git
cd agentguardbench
python scripts/generate_dataset.py
PYTHONPATH=src python -m agentguardbench --policy both
python -m unittest discover -s tests -vThe repository says evaluation results are written to results/ as machine-readable JSON and JSONL. To evaluate another policy, it recommends implementing an adapter that returns an AgentDecision and testing against the same immutable scenario set. Keep runs isolated: the repository cautions against connecting them to production credentials, real customer data, or tools capable of irreversible actions.
The project says its coverage maps to NIST AI 600-1, OWASP Agentic Top 10, OWASP MCP Top 10, and MITRE ATLAS. That is a project description of framework coverage, not evidence of independent conformance, certification, or external audit.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →What the first release does not establish
A benchmark score applies to the tested configuration and scenarios. It cannot, by itself, prove that an agent is universally safe, ethical, secure, or legally compliant. Arayemi puts the limitation plainly in the project overview: “A benchmark score describes a tested configuration. It does not prove that a system is universally safe, ethical, secure or legally compliant.”
Best Value
The author identifies several limits: template-driven cases, deterministic reference policies, translations that still need independent native-speaker review, and no repeated stochastic model trials. Planned work includes model adapters, paraphrased and indirect injections, multi-turn attacks, tool-output poisoning, context-window and memory-contamination tests, cross-tool privilege escalation, held-out challenge sets, native-speaker validation, and independent annotation with uncertainty reporting. These are plans, not capabilities established by the first release.
Real-world behavior also depends on more than the model: the deployed system prompt, memory and retrieval sources, identity controls, tool implementations, deployment context, users, monitoring, and incident procedures all affect risk. AgentGuardBench is best understood as a reproducible starting point for defensive research, education, evaluation-pipeline validation, controlled extension, and authorized security testing—not as a substitute for testing the complete deployed system.
Responsible use
Use synthetic scenarios and inert tools for development, and conduct security testing only with authorization. Do not use the benchmark as a pretext to target systems without permission, collect real personal data, bypass access controls, or automate harmful actions.
Sources: Joseph Arayemi’s 2026 project overview and the AgentGuardBench repository.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
