AgentClash is an open-source platform for evaluating AI agents on multi-turn tasks in a sandbox. It scores tool choices, cost, latency, recovery, and final results, then can preserve a failed run as a regression test for later evaluations. Each agent runs in a fresh Firecracker microVM with an isolated filesystem and network; the sandbox is removed after the run. YAML challenge packs can define tools, policy, scoring, and starting state. Available tools include file operations, data queries, HTTP, shell, and test runners. Scoring combines deterministic, mathematical, behavioral, and language-model judges with configurable weights and consensus. Provider adapters support OpenAI, Anthropic, Gemini, xAI, Mistral, and OpenRouter. CI/CD checks can run from GitHub Actions, a webhook, or the CLI, and fail builds when correctness, cost, latency, or required evidence regresses. AgentClash is MIT licensed and can be self-hosted as a full stack or used through its hosted backend. The Free plan includes 25 evaluation runs per month. Pro costs 49.00 USD per month billed monthly, or $39 / month ($468 / yr) with annual billing.

Who it is for

AgentClash suits teams evaluating multi-step AI agents for coding, research, SRE, operations, codebase questions, or support. It is also relevant to teams that want evaluation failures turned into regression checks in CI/CD.

What is good

Scores tool choices, cost, latency, recovery, and results.
Failed runs can become regression tests.
Runs agents in isolated, disposable microVMs.
Supports several model providers and OpenRouter.
Can be self-hosted or used through a hosted backend.

What to know first

Free plan allows 25 evaluation runs per month.
Pro costs 49.00 USD per month billed monthly.
Free plan requires a BYO LLM API key and sandbox token.

Verdict

AgentClash combines sandboxed agent evaluation with replayable regression tests and CI/CD checks. The Free plan has a monthly run limit; Pro costs 49.00 USD per month billed monthly or $39 / month ($468 / yr) with annual billing.

AgentClash plans and pricing

All plans

Free Free 1 workspace · 25 eval runs / month · up to 4 models per run · 7-day replay retention · BYO LLM API key · BYO E2B sandbox token · community support agentclash.dev · 1 Oct 2026

Pro $49/mo Billed monthly; annual billing $39 / month ($468 / yr) 500 eval runs / workspace / month · up to 8 models per run · 30-day replay retention · hosted sandbox with included credit · private challenge packs · CI integration · 3 concurrent eval runs · email support < 1 business day agentclash.dev · 1 Oct 2026

Team $100/mo Billed monthly; annual billing $80 / month ($960 / yr) 2,000 eval runs / workspace / month · up to 12 models per run · 90-day replay retention · 10 concurrent eval runs · multiple workspaces · workspace-level audit log · Slack notifications · priority email support < 4 business hours agentclash.dev · 1 Oct 2026

Enterprise Not published Custom SSO / SAML · org-wide audit logs · unlimited replay retention · 99.9% uptime SLA · dedicated support channel · custom MSA / billing terms agentclash.dev · 1 Oct 2026

Compared on AI agent evaluation tools

Free plan: Yes
Paid from: $39/mo
Evaluation methods: hybrid
Tool-call checks: Yes
Trace ingestion: Yes
Safety evaluations: Yes
Regression runs: Yes

Best AgentClash alternatives

See all 20

Two bars Future AGI AI Evaluation SDK Free tier No. 1$250/mo6.6 Two bars Promptfoo Free tier No. 2Free6.6 Two bars W&B Weave Free tier No. 3$60/mo6.6 Two bars DeepEval Free tier No. 4Free6.5 One bar Noveum Free tier No. 5$69/mo6.5 One bar Opik Free tier No. 6$19/mo6.5

The record

Purpose

AgentClash is an open-source AI-agent evaluation platform that runs agents on real tasks, scores outcomes, replays steps, and turns failures into regression tests.

Agent evaluation

It evaluates multi-turn agents that take actions in a real sandbox and scores tool choices, cost, latency, recovery, and the final result.

Sandboxing

Each agent runs in a fresh Firecracker microVM with an isolated filesystem and network, and the sandbox is torn down after the run.

Providers

First-class adapters support OpenAI, Anthropic, Gemini, xAI, Mistral, and OpenRouter, with more than 300 models available through OpenRouter.

Tools

Agents can use file I/O, data queries, HTTP, shell, and test runners, with declarative YAML challenge packs defining tools, policy, scoring, and starting state.

Scoring

Runs combine deterministic, mathematical, behavioural, and LLM-based judges with configurable consensus aggregation and weights.

Regression loop

When a model fails a challenge, AgentClash freezes the failing trace into a permanent test that future evaluations replay.

Integrations

CI/CD integrations can run regression tests from GitHub Actions, a webhook, or the CLI and fail builds when correctness, cost, latency, or required evidence regresses.

Security

API keys, database credentials, and OAuth tokens are stored in a scoped secret vault and injected at tool-call time without appearing in prompts, traces, or replays.

Knowledge sources

Knowledge sources include PDFs, wikis, Notion, codebases, and custom APIs, with provenance attached to retrieved facts.

Workloads

The product is positioned for coding, research, SRE, multi-step operations, codebase question answering, and support workloads.

Open source and hosting

AgentClash is MIT licensed, can be self-hosted as a full stack, or used against the hosted backend; its CLI installs from npm as the agentclash package.

Documentation

The public documentation covers the CLI, local stack, Fleet eval sets, datasets, regression gates, multi-turn human takeover, security stress harnesses, and runtime components.

Price from

$49/mo