Strands Evals is an open-source Python SDK and command-line interface for assessing AI agent behavior during development and after deployment. It scores outputs and trajectories, helps diagnose failures, checks unsafe behavior, and can simulate users and tools. Evaluators can target a single output, tool call, trace, or complete session, and multiple evaluators can be combined in an experiment. Built-in groups cover quality, safety, multimodal responses, agent behavior, and skill selection and instruction following. Deterministic checks run without an LLM judge, and developers can create domain-specific evaluators by extending the base Evaluator class. Install the package with pip as strands-agents-evals; its CLI can generate experiments, validate JSON, run evaluations, render reports, and diagnose sessions. Trace providers can retrieve data from AWS CloudWatch Logs, Langfuse, and OpenSearch. The SDK can evaluate production or staging traces without rerunning the agents. The quickstart uses Amazon Bedrock with Claude as its default judge and requires authorized credentials to invoke Claude for the example. The red-teaming API is marked experimental. The project is licensed under Apache License 2.0.
Who it is for
Strands Evals suits developers who want to validate agent behavior, measure changes, and evaluate agents through development cycles. It requires Python, and the quickstart example requires credentials authorized to invoke Claude.
What is good
- Evaluates outputs, tool calls, traces, and full sessions.
- Deterministic checks do not require an LLM judge.
- CLI supports evaluation runs and report rendering.
- Can evaluate existing production or staging traces.
What to know first
- The quickstart example requires authorized Claude credentials.
- Red-teaming API is marked experimental.
- Trace-provider access may require separate credentials or services.
Verdict
Strands Evals provides SDK and CLI workflows for agent evaluation, including deterministic checks and trace-based runs. Account for its credential and service requirements, and treat the experimental red-teaming API accordingly.
Strands Evals plans and pricing
All plansCompared on AI agent evaluation tools
- Evaluation methods
- hybrid
- Tool-call checks
- Yes
- Trace ingestion
- Yes
- Safety evaluations
- Yes
- Regression runs
- Yes
- SDK language support
- python
