Sensei

Free plan API

What Sensei is

Sensei is an open-source qualification engine for evaluating AI agents on professional tasks. It assesses task execution, reasoning, and self-improvement through feedback, then produces a score from 0 to 100 and a badge decision. Agents connect through HTTP or stdio, with documented adapters for OpenAI-compatible and LangServe connections. Evaluation uses a hybrid approach, including LLM-based and comparative judging, and supports regression runs. Sensei provides machine-readable JSON results, terminal reports, and CI/CD integration. Built-in suites cover roles such as SDR, customer support, content writing, bartender, dungeon master, and cat interview. Users can create custom suites, while the marketplace lets them discover, download, and publish community evaluations across categories including engineering, sales, marketing, product, design, support, testing, analytics, compliance, and content. The home page describes more than 70 community-built professional suites. Sensei is free, uses JavaScript SDK support, and is distributed under the MIT License.

Who it suits

Sensei suits teams and developers evaluating AI agents against standardized professional tasks, scenarios, and measurable KPIs. It is also useful for organizations that need custom suites, regression runs, reports, or CI/CD connections.

What it does well

Scores agents from 0 to 100
Supports HTTP and stdio connections
Includes built-in and custom evaluation suites
Offers marketplace community suites
Provides JSON, terminal, and CI/CD outputs

What to know first

Focused on agent evaluation rather than general use
Requires connecting an AI agent
Security policy covers version 0.x.x
JavaScript is the stated SDK language

Verdict

Sensei provides an open-source, suite-based way to measure AI-agent performance and repeat evaluations. Its adapters, reports, marketplace, and custom suites support structured qualification workflows without a paid plan.

Facts from Sensei's own pages

Agent connection: The site shows agents connecting to Sensei over HTTP or stdio. · sensei.sh, 4 Oct 2026
Built-in suites: The site lists built-in suites for SDR, customer support, content writing, bartender, dungeon master, and cat interview roles. · sensei.sh, 9 Oct 2026
CI/CD: The repository lists CI/CD integration among Sensei's capabilities. · github.com, 9 Oct 2026
Community library: The home page describes a library of 70+ community-built professional evaluation suites. · sensei.sh, 4 Oct 2026
Custom suites: Users can create their own evaluation suites and publish them to the community marketplace. · sensei.sh, 9 Oct 2026
Evaluation: It evaluates agents across task execution, reasoning, and self-improvement. · sensei.sh, 4 Oct 2026
Evaluation layers: Sensei evaluates task execution, reasoning, and self-improvement through feedback. · sensei.sh, 9 Oct 2026
Integrations: The repository documents HTTP, stdio, OpenAI-compatible, and LangServe adapters for connecting agents. · github.com, 9 Oct 2026
Intended users: Sensei is designed for people evaluating AI agents on real-world professional tasks using standardized benchmarks, scenarios, and measurable KPIs. · github.com, 9 Oct 2026
License: Sensei is distributed under the MIT License. · github.com, 9 Oct 2026
LLM judging: Sensei supports LLM-based and comparative judging, with OpenAI shown as an example provider. · github.com, 9 Oct 2026
Marketplace: The marketplace lets users discover, download, and publish AI-agent evaluation suites. · sensei.sh, 9 Oct 2026
Marketplace categories: Marketplace categories include engineering, sales, marketing, product, design, support, testing, analytics, compliance, and content. · sensei.sh, 4 Oct 2026
Purpose: Sensei is an open-source qualification engine for testing, evaluating, and certifying AI agents across professional skills. · sensei.sh, 9 Oct 2026
Reports: The repository documents machine-readable JSON results and terminal reports. · github.com, 9 Oct 2026
Scoring: Its evaluation pipeline produces a score from 0 to 100 and a badge decision. · sensei.sh, 9 Oct 2026
Security policy: The security policy supports version 0.x.x and asks vulnerability reporters to contact the maintainer by email rather than opening a public issue. · github.com, 9 Oct 2026
Security response: The policy says reports are acknowledged within 48 hours, initially assessed within one week, and targeted for a patch within 30 days of confirmation. · github.com, 9 Oct 2026
Test suites: Built-in suites cover roles including SDR, customer support, content writer, bartender, dungeon master, and cat interview. · sensei.sh, 4 Oct 2026

Where Sensei stands on Freedom251

AI Agent Evaluation Tools#24 of 28