Promptfoo

Promptfoo is an AI evaluation and red-teaming tool for developers and organizations building AI applications. It is best suited to teams that want to compare model outputs and probe applications for security risks in the same workflow. Its free Community plan is broad, but a monthly red-team probe cap and Node.js requirement shape its fit.

Overview

Promptfoo combines prompt evaluation across models with simulated-user testing for weaknesses in AI systems, including agents. Founded in 2024, the San Francisco-based company says it is now part of OpenAI and remains open source. Community can run locally or be self-hosted; larger teams can choose managed cloud or on-premise deployment.

Key features

Evaluation across models

Promptfoo compares prompt outputs from multiple models in a web view, making it useful for teams assessing alternatives without splitting the work across provider-specific tools. Its getting-started guide describes support for more than 60 providers, including OpenAI, Anthropic, Google, and local models such as Ollama.

Community includes all LLM evaluation features and providers and integrations, as well as custom metrics, LLM-as-a-judge, safety evaluations, human review workflows, and CI/CD integration. Listed connections include GitHub Actions, GitLab CI, Jenkins, Azure Pipelines, CircleCI, Bitbucket Pipelines, Splunk, Burp Suite, Google Sheets, and MCP. That range can connect evaluation with build and security processes, though the value depends on whether a team’s existing workflow uses those tools.

Red teaming

Red-team runs simulate users to seek prompt injections, jailbreaks, data leaks, business-rule violations, and unsafe agent tool use. This gives Promptfoo a security-testing role alongside evaluation, rather than limiting it to comparing output quality. Community’s 10,000 monthly probes are meaningful for regular testing, but teams running frequent or broad campaigns may need custom limits through Enterprise or On-Premise. A probe is one request to the target system.

Installation and runtime

Promptfoo installs through npm, npx, or Homebrew; Homebrew supports Mac and Linux. It requires Node.js 22.22.0 or newer and recommends Node.js 24 LTS. The multiple install routes are useful, but the Node.js minimum is a practical constraint for teams whose environments cannot adopt that runtime.

Pricing

Community is the clear starting point for individuals and teams that can work within the monthly probe allowance and community support. Enterprise is aimed at organizations that need shared workflows, monitoring, identity controls, managed hosting, or contractual support. On-Premise is the fit when infrastructure control and data isolation are priorities. Both paid plans use custom pricing, so neither offers a fixed entry cost.

Platforms

Promptfoo supports API, Linux, macOS, self-hosted, web, and Windows. Its deployment choices span local or self-hosted Community use, managed cloud Enterprise, and customer-infrastructure On-Premise deployment. The site displays SOC 2 and ISO 27001 certification badges.

Who it's for

Promptfoo is a strong fit for developers and AI teams that need to compare prompts across hosted and local models, add evaluation to CI/CD, and look for security failures in agents or other AI applications. Its broad provider and integration support suits mixed-model workflows. It is less compelling for teams seeking a fixed-price managed service, unlimited red-team usage, or a runtime-neutral installation.

Pros and cons

• Pros: Evaluation and red teaming sit in one workflow, so teams can assess output quality and probe for security risks without treating them as separate tasks.
• Pros: More than 60 providers, named CI/CD and security integrations, and support for local models give teams options across varied AI stacks.
• Pros: The free plan includes all LLM evaluation features and integrations, with local and self-hosted deployment.
• Cons: Community red teaming is capped at 10,000 probes per month, which can constrain frequent testing.
• Cons: The Node.js 22.22.0 minimum may complicate adoption in environments that cannot use that runtime.
• Cons: Enterprise and On-Premise use custom pricing, making cost harder to compare before discussing team needs.

Alternatives

Future AGI AI Evaluation SDK is worth considering for teams seeking an Apache 2.0 platform with Docker, Python SDK, and Node SDK self-hosting options.

W&B Weave may suit teams wanting evaluations, tracing, and scorers with a free plan that specifies monthly ingestion and storage allowances.

DeepEval is an alternative for teams seeking an Apache 2.0 open-source evaluation framework with a local and CI/CD test runner.

Noveum is another freemium option for teams whose requirements fit its stated free-plan credits, spans, storage, member, and retention limits.

Opik offers an open-source core observability and evaluation feature set that users can download and run locally.

Strands Evals is a free open-source Python SDK and CLI option; model-provider and trace-provider access may require separate credentials.

AgentClash may fit teams looking for a hosted free tier with defined workspace, run, model, and replay-retention limits.

Arklex is another option to consider.

Browse AI Agent Evaluation Tools, AI Prompt Management Software, LLM Evaluation Tools, AI Red Teaming Tools, AI Security Testing Tools, and AI LLM Evaluation Tools for more options.

Verdict

Choose Promptfoo if your team needs one tool for cross-model evaluation and red-team testing, especially when local models, CI/CD connections, or self-hosting matter. Its generous feature set at no cost makes it a practical starting point; look elsewhere if the 10,000-probe monthly ceiling, Node.js requirement, or custom-priced paid deployment does not suit your needs.