Freedom251 › AI LLM Evaluation Tools
OpenAI Evals

What OpenAI Evals is
OpenAI Evals is an open-source platform for checking model outputs against criteria you define. Each evaluation uses a dataset representing the inputs a prompt should handle, a data-source schema, and graders that judge whether outputs meet the criteria. Grader types include string checks, text similarity, score models, label models, and Python code execution. You can configure runs in the OpenAI Dashboard or manage them through the Evals API. Runs are asynchronous, support larger data volumes, and allow performance tracking across model versions. Evaluations can target OpenAI models, listed third-party providers, or custom HTTPS endpoints. Third-party options include Google, Anthropic on AWS Bedrock, Together, and Fireworks. Private evaluations can use your own data without public exposure. The package can be installed locally with pip and requires Python 3.9 or newer. Integrations include Snowflake logging and Weights & Biases. Running evaluations requires an OpenAI API key and incurs model-usage costs. External models do not support tool calls. The service is scheduled to become read-only on October 31, 2026, and shut down on November 30, 2026.
Who it suits
It suits developers and teams defining repeatable checks for model quality, safety, style, or content. Users needing third-party models must meet the stated organization tier and administrator requirements.
What it does well
- Dashboard and API configuration options
- Multiple grader types, including Python execution
- Supports OpenAI, third-party, and custom endpoints
- Asynchronous runs track performance across versions
- Private evaluations can use internal data
What to know first
- API key and model-usage costs are required
- External-model tool calls are unsupported
- Third-party models require eligible organization approval
- Scheduled to become read-only and then shut down
Limits
Running evals requires an OpenAI API key and generates model-usage charges. External-model runs do not support tool calls, and third-party models require an organization at usage tier 1 or higher, admin enablement, and acceptance of a usage disclaimer. The platform is scheduled to become read-only on October 31, 2026, and shut down on November 30, 2026.
Getting started
- Open the OpenAI Evals website or Dashboard
- Prepare a test dataset representing your expected prompt inputs
- Define the data-source schema and evaluation criteria
- Choose graders such as string, similarity, model, label, or Python graders
- Configure the evaluation in the Dashboard or through the Evals API
- Provide an OpenAI API key and run the evaluation
Verdict
OpenAI Evals offers configurable dataset-based testing with several grader types and deployment choices. Its announced read-only and shutdown dates are important for anyone planning a longer-term workflow.
Questions
Is OpenAI Evals free?
The pricing model is free and the project is open source, but evaluation runs incur API costs for model usage.
Which platforms does it support?
It supports API, self-hosted, and web use. The package can also be installed locally with pip on Python 3.9 or newer.
Can it evaluate third-party models?
Yes. Listed providers are Google, Anthropic hosted on AWS Bedrock, Together, and Fireworks, along with custom model endpoints.
What is required for third-party models?
You need an OpenAI organization at usage tier 1 or higher, an administrator to enable the feature, and acceptance of its usage disclaimer.
Can evaluations remain private?
Yes. Users can build private evaluations from their own data without exposing that data publicly.
When will Evals be discontinued?
OpenAI says Evals becomes read-only for existing users on October 31, 2026, and is scheduled to shut down on November 30, 2026.
Facts from OpenAI Evals's own pages
- Alternative
- OpenAI recommends trying Datasets for a more iterative environment to experiment while building an eval. · developers.openai.com, 30 Sept 2026
- API access
- Evals can be created and managed programmatically through the Evals API or configured in the dashboard. · developers.openai.com, 1 Oct 2026
- Asynchronous scale
- Evals run asynchronously, support larger data volumes, and let users monitor performance across versions. · developers.openai.com, 1 Oct 2026
- Custom endpoint security
- Custom model endpoints must use HTTPS, and OpenAI says it encrypts API keys supplied for them. · developers.openai.com, 30 Sept 2026
- Dashboard
- Evals can be configured and run directly in the OpenAI Dashboard. · github.com, 1 Oct 2026
- Dashboard and API
- You can configure evals in the OpenAI dashboard or programmatically with the Evals API. · developers.openai.com, 30 Sept 2026
- Data handling
- Calls to external models pass data to third parties and are subject to different terms and weaker safety guarantees than calls to OpenAI models. · developers.openai.com, 30 Sept 2026
- Data residency
- The /v1/evals endpoint supports United States and Europe (EEA plus Switzerland) data residency, with service-level support listed. · developers.openai.com, 1 Oct 2026
- Deprecation
- OpenAI says the Evals platform becomes read-only for existing users on October 31, 2026, and the dashboard and API are scheduled to shut down on November 30, 2026. · developers.openai.com, 1 Oct 2026
- Evaluation criteria
- An eval uses a data-source schema and testing criteria consisting of graders that determine whether model output is correct. · developers.openai.com, 1 Oct 2026
- External model limits
- Tool calls are currently not supported when running evals with external models. · developers.openai.com, 30 Sept 2026
- External models
- The OpenAI Platform can run evals against third-party models and custom model endpoints. · developers.openai.com, 1 Oct 2026
- Graders
- Supported grader types include string check, text similarity, score model, label model, and Python code execution graders. · developers.openai.com, 1 Oct 2026
- Integrations
- The repository supports logging eval results to Snowflake and says evals can be run and created using Weights & Biases. · github.com, 1 Oct 2026
- Local installation
- The open-source package can be installed locally with pip, and the repository requires Python 3.9 or newer. · github.com, 1 Oct 2026
- Model support
- Evaluations can use OpenAI models, third-party models, and custom model endpoints. · developers.openai.com, 30 Sept 2026
- Private evaluations
- Users can build private evals from their own data without exposing that data publicly. · github.com, 1 Oct 2026
- Purpose
- Evals is a framework for evaluating large language models and LLM systems, with an open-source benchmark registry. · github.com, 1 Oct 2026
- Security compliance
- OpenAI states that its API business services have undergone an independent SOC 2 Type 2 examination and that it maintains ISO/IEC 27001:2022 and ISO/IEC 27701:2019 certifications for supporting systems. · openai.com, 1 Oct 2026
- Test data
- An eval run uses a test dataset representing the kind of data you expect your prompt to handle. · developers.openai.com, 30 Sept 2026
- Third-party eligibility
- Using third-party models requires an OpenAI organization at usage tier 1 or higher and an admin to enable the feature and accept its usage disclaimer. · developers.openai.com, 30 Sept 2026
- Third-party providers
- The listed third-party model providers are Google, Anthropic hosted on AWS Bedrock, Together, and Fireworks. · developers.openai.com, 30 Sept 2026
- Third-party spend limits
- OpenAI currently covers third-party inference costs up to monthly limits of $5, $25, $50, $100, and $200 for usage tiers 1 through 5, respectively. · developers.openai.com, 30 Sept 2026
- Usage cost
- Running evals requires an OpenAI API key and incurs the API costs associated with model usage. · github.com, 1 Oct 2026
Where OpenAI Evals stands on Freedom251
- AI LLM Evaluation Tools#28 of 30