Inspect AI

What Inspect AI is

Inspect AI is an open-source framework for evaluating large language models across coding, agent tasks, reasoning, knowledge, behavior, and multimodal understanding. Teams can build evaluations from datasets, solvers, and scorers or choose from more than 200 pre-built evaluations. Inspect View presents evaluation activity and results in a web interface, and a VS Code extension helps users browse logs, run and debug tasks, and adjust options.

The framework connects to more than 20 model providers and supports local inference with Hugging Face, vLLM, and SGLang. Provider groups include OpenAI, Anthropic, Google, AWS Bedrock, Azure AI, Hugging Face, Ollama, and others. Included tools cover web search, code execution, text editing, computer use, and web browsing; MCP tools are also supported. Teams can connect agents made with OpenAI Agents SDK, Pydantic AI, or LangChain, along with CLI agents such as Claude Code, Codex CLI, and Gemini CLI. Untrusted model code can run in Docker, Kubernetes, Modal, Proxmox, Vagrant, or other systems through an extension API. Inspect is self-hosted and free; getting started uses pip install inspect-ai, and evaluation development and runs require access to a model.

Who it suits

Inspect AI suits developers and teams building evaluations for language models, including coding, reasoning, behavior, and multimodal tasks. It is a fit for organizations that want self-hosted evaluation, model-provider and agent integrations, and tools for inspecting results.

Getting started

1. Open https://inspect.aisi.org.uk/
2. Install Inspect from PyPI with pip install inspect-ai
3. Arrange access to a model for evaluation development and runs
4. Use the framework to develop or run evaluations
5. Review and visualize evaluation activity in Inspect View or browse logs in the VS Code extension

Verdict

Choose Inspect AI if you need a free, self-hosted framework to evaluate language models across varied task types, with broad provider and agent support. Look elsewhere if you need evaluation runs without access to a model or a hosted deployment.