AgentClash

Free plan APISelf-hostedWeb

What AgentClash is

AgentClash is an open-source platform for evaluating AI agents on multi-turn tasks in a sandbox. It scores tool choices, cost, latency, recovery, and final results, then can preserve a failed run as a regression test for later evaluations. Each agent runs in a fresh Firecracker microVM with an isolated filesystem and network; the sandbox is removed after the run. YAML challenge packs can define tools, policy, scoring, and starting state. Available tools include file operations, data queries, HTTP, shell, and test runners. Scoring combines deterministic, mathematical, behavioral, and language-model judges with configurable weights and consensus. Provider adapters support OpenAI, Anthropic, Gemini, xAI, Mistral, and OpenRouter. CI/CD checks can run from GitHub Actions, a webhook, or the CLI, and fail builds when correctness, cost, latency, or required evidence regresses. AgentClash is MIT licensed and can be self-hosted as a full stack or used through its hosted backend. The Free plan includes 25 evaluation runs per month. Pro costs $49 a month billed monthly, or $39 / month ($468 / yr) with annual billing.

Who it suits

AgentClash suits teams evaluating multi-step AI agents for coding, research, SRE, operations, codebase questions, or support. It is also relevant to teams that want evaluation failures turned into regression checks in CI/CD.

What it does well

Scores tool choices, cost, latency, recovery, and results.
Failed runs can become regression tests.
Runs agents in isolated, disposable microVMs.
Supports several model providers and OpenRouter.
Can be self-hosted or used through a hosted backend.

What to know first

Free plan allows 25 evaluation runs per month.
Pro costs $49 a month billed monthly.
Free plan requires a BYO LLM API key and sandbox token.

Limits

The Free plan includes 25 evaluation runs per month, up to 4 models per run, 7-day replay retention, one workspace, and community support. It requires your own LLM API key and E2B sandbox token.

Plans

Plan	Billed and limits	Price
Free	1 workspace · 25 eval runs / month · up to 4 models per run · 7-day replay retention · BYO LLM API key · BYO E2B sandbox token · community support	Free
Team	Billed monthly; annual billing $80 / month ($960 / yr) · 2,000 eval runs / workspace / month · up to 12 models per run · 90-day replay retention · 10 concurrent eval runs · multiple workspaces · workspace-level audit log · Slack notifications · priority email support < 4 business hours	$100 a month
Pro	Billed monthly; annual billing $39 / month ($468 / yr) · 500 eval runs / workspace / month · up to 8 models per run · 30-day replay retention · hosted sandbox with included credit · private challenge packs · CI integration · 3 concurrent eval runs · email support < 1 business day	$49 a month
Enterprise	Custom · SSO / SAML · org-wide audit logs · unlimited replay retention · 99.9% uptime SLA · dedicated support channel · custom MSA / billing terms	On request

As agentclash.dev lists them, read 1 Oct 2026. The maker's own page has the final word.

Getting started

Visit https://www.agentclash.dev/ and choose the hosted backend or self-hosted full stack.
For the CLI route, install the agentclash package from npm.
Choose the Free plan or a paid plan that fits your evaluation volume.
For the Free plan, provide your own LLM API key and E2B sandbox token.
Define a challenge pack in YAML and run an evaluation through the CLI or a supported integration.

Verdict

AgentClash combines sandboxed agent evaluation with replayable regression tests and CI/CD checks. The Free plan has a monthly run limit; Pro costs $49 a month billed monthly or $39 / month ($468 / yr) with annual billing.

Questions

Is AgentClash free?

Yes. The Free plan costs nothing and includes 25 evaluation runs per month.

How much do paid plans cost?

Pro is $49 a month when billed monthly, or $39 / month ($468 / yr) with annual billing. Team is $100 a month when billed monthly, or $80 / month ($960 / yr) with annual billing. Enterprise pricing is custom.

Can I host AgentClash myself?

Yes. AgentClash is MIT licensed and can be self-hosted as a full stack, or used with its hosted backend.

What platforms does it support?

AgentClash lists API, self-hosted, and web platforms. Its CLI installs from npm as the agentclash package.

Which model providers does it support?

First-class adapters support OpenAI, Anthropic, Gemini, xAI, Mistral, and OpenRouter. More than 300 models are available through OpenRouter.

What does the Free plan include?

It includes one workspace, 25 evaluation runs per month, up to 4 models per run, and 7-day replay retention. It requires your own LLM API key and E2B sandbox token.

Facts from AgentClash's own pages

Agent evaluation: It evaluates multi-turn agents that take actions in a real sandbox and scores tool choices, cost, latency, recovery, and the final result. · agentclash.dev, 1 Oct 2026
Documentation: The public documentation covers the CLI, local stack, Fleet eval sets, datasets, regression gates, multi-turn human takeover, security stress harnesses, and runtime components. · agentclash.dev, 1 Oct 2026
Integrations: CI/CD integrations can run regression tests from GitHub Actions, a webhook, or the CLI and fail builds when correctness, cost, latency, or required evidence regresses. · agentclash.dev, 1 Oct 2026
Knowledge sources: Knowledge sources include PDFs, wikis, Notion, codebases, and custom APIs, with provenance attached to retrieved facts. · agentclash.dev, 1 Oct 2026
Open source and hosting: AgentClash is MIT licensed, can be self-hosted as a full stack, or used against the hosted backend; its CLI installs from npm as the agentclash package. · agentclash.dev, 1 Oct 2026
Providers: First-class adapters support OpenAI, Anthropic, Gemini, xAI, Mistral, and OpenRouter, with more than 300 models available through OpenRouter. · agentclash.dev, 1 Oct 2026
Purpose: AgentClash is an open-source AI-agent evaluation platform that runs agents on real tasks, scores outcomes, replays steps, and turns failures into regression tests. · agentclash.dev, 1 Oct 2026
Regression loop: When a model fails a challenge, AgentClash freezes the failing trace into a permanent test that future evaluations replay. · agentclash.dev, 1 Oct 2026
Sandboxing: Each agent runs in a fresh Firecracker microVM with an isolated filesystem and network, and the sandbox is torn down after the run. · agentclash.dev, 1 Oct 2026
Scoring: Runs combine deterministic, mathematical, behavioural, and LLM-based judges with configurable consensus aggregation and weights. · agentclash.dev, 1 Oct 2026
Security: API keys, database credentials, and OAuth tokens are stored in a scoped secret vault and injected at tool-call time without appearing in prompts, traces, or replays. · agentclash.dev, 1 Oct 2026
Tools: Agents can use file I/O, data queries, HTTP, shell, and test runners, with declarative YAML challenge packs defining tools, policy, scoring, and starting state. · agentclash.dev, 1 Oct 2026
Workloads: The product is positioned for coding, research, SRE, multi-step operations, codebase question answering, and support workloads. · agentclash.dev, 1 Oct 2026

Where AgentClash stands on Freedom251

AI Agent Evaluation Tools#9 of 28