PromptPex

Paid LinuxMacSelf-hostedWebWindows

What PromptPex is

PromptPex is an open-source tool for developers who keep prompts in their codebase and want to examine how models follow prompt requirements. It treats prompts as functions, identifies output rules expressed in natural language, and creates unit tests for checking those rules. It can evaluate prompt performance across models, using an LLM to assess whether responses meet specified requirements. Test generation can be shaped with scenarios, expansions, custom instructions, and effort levels. Its CLI accepts Markdown or text templates, Prompty files, and GitHub Models prompt YAML files, and is available through npm or Docker. Setup documentation lists OpenAI, Azure OpenAI, GitHub Models, and Ollama providers. Integrations include OpenAI Evals, Azure OpenAI Stored Completions, GitHub Models Evals, and GitHub Actions; a local web interface is also described. Microsoft is identified as the maker. The project is intended for research, and the maker advises further evaluation before downstream use. Vague prompts may lead to weak results, and rule extraction can vary in accuracy, particularly for complex or idiomatic wording.

Who it suits

PromptPex suits developers who keep AI prompts in a codebase and want generated tests of prompt requirements. It is described as a research release, not as a ready-made downstream application.

What it does well

Generates unit tests from prompt output rules
Supports several model providers, including Ollama
Offers CLI and Docker use
Documents integrations for evaluation tools and GitHub Actions

What to know first

Intended for research, not downstream use without further evaluation
Vague prompts can produce suboptimal results
Rule extraction accuracy can vary with complex language

Limits

The tool is intended for research, and the maker advises evaluating robustness, safety, harm, and bias before downstream use. Vague prompts can yield suboptimal results, while extraction accuracy can vary for complex or idiomatic language.

Getting started

Visit the PromptPex project website.
Install or run the npm command-line tool, or use Docker.
Choose a supported provider such as OpenAI, Azure OpenAI, GitHub Models, or Ollama.
Provide the appropriate provider credentials.
Use a Markdown or text template, Prompty file, or GitHub Models prompt YAML file.
Configure scenarios, test expansions, custom instructions, or effort levels as needed.

Verdict

PromptPex offers a way to generate and evaluate tests for prompt requirements across models. Its research status and stated extraction limitations make additional evaluation important before relying on results.

Questions

Is PromptPex free?

Yes. It is listed as free and open source.

Who makes PromptPex?

Microsoft is identified as its maker.

Which model providers does it support?

Setup materials list OpenAI, Azure OpenAI, GitHub Models, and Ollama.

How can PromptPex be run?

It is packaged as an npm command-line tool and can also run in Docker. A local web interface is also described.

What integrations are documented?

They include OpenAI Evals, Azure OpenAI Stored Completions, GitHub Models Evals, and GitHub Actions.

Is PromptPex intended for production applications?

The released library is intended for research. The maker advises additional evaluation of robustness, safety, harm, and bias before downstream use.

Facts from PromptPex's own pages

Automatic rule extraction: PromptPex extracts output rules written in natural language in a prompt, such as a requirement to format output as JSON. · github.com, 8 Oct 2026
CLI and Docker: PromptPex is packaged as an npm command-line tool and can also be run in Docker. · microsoft.github.io, 8 Oct 2026
Configurable generation: Test generation can be configured with scenarios, test expansions, and custom instructions. · microsoft.github.io, 8 Oct 2026
Deployment: The project README gives a Docker command for running PromptPex, and the setup guide describes a local web interface. · github.com, 8 Oct 2026
Feedback: The transparency note invites feedback and collaboration by email and says the team will update the repository with mitigations for reported or identified issues. · microsoft.github.io, 8 Oct 2026
Integrations: PromptPex documents exports and integrations for OpenAI Evals, Azure OpenAI Stored Completions, GitHub Models Evals, and GitHub Actions. · microsoft.github.io, 8 Oct 2026
Intended audience: The project describes its intended users as developers who keep prompts in their codebase. · microsoft.github.io, 8 Oct 2026
Intended use: The released open-source library is intended for research, and the maker says it should not be used in downstream applications without additional evaluation of robustness, safety, harm, and bias. · microsoft.github.io, 8 Oct 2026
Limitations: The maker says vague prompts can produce suboptimal results and rule extraction can vary in accuracy, especially with complex or idiomatic language. · microsoft.github.io, 8 Oct 2026
Maker: The documentation footer identifies Microsoft as the maker of PromptPex. · microsoft.github.io, 8 Oct 2026
Model evaluation: PromptPex can evaluate prompt performance across models, and uses an LLM to assess whether model outputs meet specified requirements. · github.com, 8 Oct 2026
Model providers: The setup guide says users can configure OpenAI, Azure OpenAI, or Ollama credentials. · microsoft.github.io, 8 Oct 2026
Prompt formats: The CLI supports Markdown or text templates, Prompty files, and GitHub Models prompt YAML files. · microsoft.github.io, 8 Oct 2026
Purpose: PromptPex is a tool for developers to explore and test AI model prompts by treating prompts as functions and generating test inputs. · microsoft.github.io, 8 Oct 2026
Rule extraction: It can extract output rules written in natural language in a prompt. · github.com, 8 Oct 2026
Support: The transparency note invites feedback and reports of unexpected or offensive behavior by email to listed project contacts. · microsoft.github.io, 8 Oct 2026
Supported model providers: The CLI setup lists OpenAI, Azure OpenAI, GitHub Models, and Ollama among supported LLM providers. · microsoft.github.io, 8 Oct 2026
Test generation: It generates unit test cases designed to check whether a prompt makes a model follow its output rules. · github.com, 8 Oct 2026
Test generation controls: Generation can be configured with scenarios, test expansions, custom instructions, and effort levels. · microsoft.github.io, 8 Oct 2026

Where PromptPex stands on Freedom251

AI Prompt Generators#23 of 28