Exgentic

Paid APILinuxSelf-hostedWeb

What Exgentic is

Exgentic is an open-source leaderboard and evaluation framework for comparing general-purpose AI agents across benchmarks. It provides a shared interface for running evaluations in areas such as personal assistance, customer service, technical support, deep research, and software engineering. Listed benchmarks include τ²-bench, AppWorld, BrowseComp+, SWE-Bench, HotpotQA, GSM8K, and the Berkeley Function Calling Leaderboard. Listed agents include LiteLLM Tool Calling, SmolAgents, OpenAI MCP, Claude Code, Codex CLI, and Gemini CLI. Exgentic says it avoids prompt optimization, samples 100 tasks per benchmark, and leaves benchmark content and agent implementations unchanged apart from interface adaptation for its protocol. Pipelines, implementations, configurations, and results are available for reproduction, although outcomes may vary with model versions or nondeterministic output. Users can install a command-line tool or Python library, use an evaluation dashboard, and connect OpenAI, Anthropic, or Hugging Face models. A Docker runner provides container isolation. The project is free under Apache License 2.0 and directs support questions to GitHub issues.

Who it suits

Exgentic is intended for agent builders, researchers, component developers, benchmark builders, and general users interested in reproducible agent comparisons. Python users can run it locally or evaluate supported model integrations.

What it does well

Shared interface across diverse agent benchmarks
Open evaluation pipeline, configurations, and results
Supports OpenAI, Anthropic, and Hugging Face models
Command-line and Python installation options
Docker runner for container isolation

What to know first

Results may vary with model versions
Nondeterministic outputs can change results
Docker isolation requires Docker installed and running
Evaluation sessions stop at 100 steps or actions

Limits

Evaluation sessions default to 100 steps and 100 actions, ending when either threshold is reached. Results can vary with model versions or nondeterministic outputs.

Getting started

Open the Exgentic website or repository
Install the command-line tool with uv
Install the Python library with uv or pip
Provide OpenAI or Anthropic API credentials, or use Hugging Face models or Jobs
Install and run Docker for full container isolation
Use the evaluation dashboard command to review runs

Verdict

Exgentic offers an open, reproducible way to compare agents across named benchmarks and domains. Its published limits and possible result variation should be considered when interpreting evaluations.

Questions

Is Exgentic free?

Yes. Exgentic is free and open source under the Apache License 2.0.

Which platforms does Exgentic support?

It supports API, Linux, self-hosted, and web use.

Which agents and benchmarks are available?

Listed agents include LiteLLM Tool Calling, SmolAgents, OpenAI MCP, Claude Code, Codex CLI, and Gemini CLI. Benchmarks include τ²-bench, AppWorld, BrowseComp+, SWE-Bench, HotpotQA, GSM8K, and Berkeley Function Calling Leaderboard.

Can evaluations be reproduced?

The repository provides the evaluation pipeline, agent implementations, configuration, and results. Exgentic notes that model versions and nondeterministic outputs can affect results.

What integrations are documented?

The quick start documents OpenAI and Anthropic credentials, while the repository also describes Hugging Face models and Hugging Face Jobs.

Facts from Exgentic's own pages

Agents: Listed agents include LiteLLM Tool Calling, SmolAgents, OpenAI MCP, Claude Code, Codex CLI, and Gemini CLI. · github.com, 4 Oct 2026
Available agents: The listed agents include LiteLLM Tool Calling, HuggingFace SmolAgents, OpenAI MCP, Claude Code, Codex CLI, and Gemini CLI. · github.com, 9 Oct 2026
Available benchmarks: The framework lists tau2, appworld, browsecompplus, swebench, hotpotqa, gsm8k, and bfcl benchmarks. · github.com, 9 Oct 2026
Benchmark handling: Exgentic says it does not modify benchmarks or agent implementations, though it may adapt interfaces to its unified protocol. · exgentic.ai, 4 Oct 2026
Benchmarks: The leaderboard covers personal assistance, customer service, technical support, deep research, and software engineering benchmarks. · exgentic.ai, 9 Oct 2026
Evaluation approach: Exgentic says it avoids prompt optimization and reports results on 100 sampled tasks per benchmark. · exgentic.ai, 9 Oct 2026
Evaluation domains: The leaderboard covers personal assistance, customer service, technical support, deep research, and software engineering tasks. · exgentic.ai, 4 Oct 2026
Evaluation framework: Its framework provides a consistent interface for evaluating agents on benchmarks to compare performance and reproduce results. · github.com, 4 Oct 2026
Framework: Exgentic provides a consistent interface for testing agents across benchmarks and domains to compare performance and reproduce results. · github.com, 9 Oct 2026
Installation: The project documents installation as a command line tool with uv and use as a Python library with uv or pip. · github.com, 4 Oct 2026
Integrations: The README documents API credentials for OpenAI and Anthropic models and support for Hugging Face models and Jobs. · github.com, 9 Oct 2026
Isolation: The repository documents a Docker runner for full container isolation, requiring Docker to be installed and running. · github.com, 4 Oct 2026
License: The framework is distributed under the Apache License 2.0. · github.com, 9 Oct 2026
License and support: The repository states that Exgentic is licensed under Apache License 2.0 and directs support questions to GitHub issues. · github.com, 4 Oct 2026
Local use: The README documents installation as a command-line tool or Python library and includes an evaluation dashboard command. · github.com, 9 Oct 2026
Methodology: Exgentic says it avoids prompt optimization and reports results on 100 sampled tasks per benchmark. · exgentic.ai, 4 Oct 2026
Model integrations: The quick start documents OpenAI and Anthropic API credentials, and the repository also describes support for Hugging Face models or evaluations on Hugging Face Jobs. · github.com, 4 Oct 2026
Open results: The evaluation pipeline, agent implementations, configuration, and results are available for reproducing evaluations. · github.com, 9 Oct 2026
Product: Exgentic is an open agent leaderboard that compares general purpose AI agents across diverse benchmarks. · exgentic.ai, 4 Oct 2026
Purpose: Exgentic is an open agent leaderboard that evaluates general purpose AI agents across diverse domains. · exgentic.ai, 9 Oct 2026
Reproducibility: The evaluation pipeline, agent implementations, and configuration are available in the repository; the project notes results may vary with model versions or nondeterministic outputs. · exgentic.ai, 4 Oct 2026
Run limits: Evaluation sessions default to limits of 100 steps and 100 actions, and stop when either limit is reached. · github.com, 9 Oct 2026
Security and isolation: The README documents a Docker runner for full container isolation when running evaluations. · github.com, 9 Oct 2026
Support: The project directs users to open a GitHub issue for questions and support. · github.com, 9 Oct 2026

Where Exgentic stands on Freedom251

AI Agent Evaluation Tools#27 of 28