Pythia

Paid Self-hosted

What Pythia is

Pythia is a free, open-source, self-hosted engine for automated prompt optimization. It tests prompts against user-provided datasets, evaluates them on held-out data, studies aggregated failure patterns, and proposes ranked rewrites. A controller can continue iterating, return to an earlier stronger prompt, or stop when results converge. The system records prompt versions, scores, and controller decisions, so users can compare runs, export results, and roll back to checkpoints. Pythia uses LangGraph stateful agent workflows and is described as locally hosted and privacy-preserving, with optimization running on open-source models within an institution. It supports models from one billion to one trillion parameters. Gemini and Ollama backends are included, and another backend can be used when its LLM provides an invoke() method. CSV files are recommended, with visit and Ground Truth columns.

Who it suits

Pythia suits teams optimizing prompts against structured development and validation datasets, especially clinical users balancing sensitivity and specificity. It can also support general classification examples such as identifying prospective home buyers.

What it does well

Free and open source
Self-hosted, privacy-preserving workflow
Held-out evaluation with configurable metrics
Prompt history, comparison, export, and rollback
Gemini and Ollama backends included

What to know first

API-call heavy workflow may increase costs
Requires development and validation datasets
Documented CSV format uses specific columns
Self-hosted setup is required

Verdict

Pythia provides a structured loop for testing, diagnosing, rewriting, and tracking prompts. Its local hosting, configurable clinical metrics, and checkpoint history fit organizations that can supply the required datasets and infrastructure.

Facts from Pythia's own pages

Backends: The repository README says Pythia includes Gemini and Ollama backends and can use another backend whose LLM is called with an .invoke() method. · github.com, 5 Oct 2026
Clinical focus: The project describes its optimization as clinical and sensitive to sensitivity and specificity. · clai-group.github.io, 11 Oct 2026
Configurable thresholds: The README says sensitivity and specificity thresholds are configurable and default to 0.75, while the priority metric defaults to specificity. · github.com, 5 Oct 2026
Data requirements: The documented workflow requires development and validation datasets, an output directory, an SOP, and an initial prompt. · github.com, 11 Oct 2026
Dataset format: The README recommends CSV datasets, with each CSV representing a person and rows representing visits or notes, using “visit” and “Ground Truth” columns. · github.com, 5 Oct 2026
Error analysis: A specialist LLM analyzes failures in aggregate, identifies root causes, and proposes improvement hypotheses. · clai-group.github.io, 11 Oct 2026
Evaluation: Pythia evaluates prompts against a held-out dataset using configurable metrics and tracks results across iterations. · clai-group.github.io, 11 Oct 2026
Framework: The site says Pythia uses LangGraph stateful agentic workflows. · clai-group.github.io, 5 Oct 2026
History and export: Pythia logs prompt versions, scores, and controller decisions, and the site says users can compare runs, export results, and roll back to earlier checkpoints. · clai-group.github.io, 5 Oct 2026
Hosting and privacy: The project describes Pythia as locally hosted and privacy-preserving, with optimization running on open-source models within the institution. · clai-group.github.io, 11 Oct 2026
Intended users: The project presents Pythia for clinical prompt optimization and also gives a general example involving home-buying inquiries. · github.com, 11 Oct 2026
Iteration control: A controller can continue, backtrack to a stronger prior prompt, or halt when the prompt has converged. · clai-group.github.io, 11 Oct 2026
Local hosting: The maker describes Pythia as locally hosted and privacy-preserving, with optimization running on open-source models within the institution. · clai-group.github.io, 5 Oct 2026
Model backends: The repository says Pythia includes Gemini and Ollama backends and can use another LLM backend that provides an invoke() method. · github.com, 11 Oct 2026
Model scale: The site says optimization can run on models ranging from one billion to one trillion parameters. · clai-group.github.io, 5 Oct 2026
Prompt improvements: An improver agent generates and ranks multiple candidate prompt rewrites before evaluation. · clai-group.github.io, 11 Oct 2026
Purpose: Pythia is an automated prompt optimization engine that iteratively evaluates prompts, analyzes failures, and synthesizes improvements. · clai-group.github.io, 11 Oct 2026
Run history: Pythia logs prompt versions, scores, and controller decisions, and supports comparing runs, exporting results, and rolling back to checkpoints. · clai-group.github.io, 11 Oct 2026
Support: The project links to its GitHub repository issues for questions and issue reporting. · clai-group.github.io, 11 Oct 2026
Usage limit: The repository warns that Pythia is API-call heavy and says fewer iterations or a smaller dataset can reduce token costs and computation. · github.com, 11 Oct 2026
Workflow: Its workflow evaluates prompts on a held-out dataset, analyzes failure patterns, proposes prompt rewrites, and controls whether to continue, backtrack, or stop. · clai-group.github.io, 5 Oct 2026

Where Pythia stands on Freedom251

AI Prompt Generators#27 of 28