SuperRed

Free plan Linux

What SuperRed is

SuperRed is a free, open-source Python framework for red-teaming chatbots, agents and assistants. It tests whether attacks can violate defined security properties by coordinating interchangeable attacker, system-under-test and benchmark modules. Runs can set the attacker model, per-task spending limit, trust-boundary access and benchmark-feedback visibility. Its catalogue contains 35 modules: 21 attackers, 6 targets and 8 benchmarks, including PAIR, TAP, AutoDAN-Turbo, Crescendo, AgentVigil, HarmBench, AgentDojo and DecodingTrust-Agent. Targets may represent fixed-response fixtures, simulated environments, sandboxes or live deployments. Runs record model, cost and success rate, with live terminal progress and web reports containing task details and trajectories. Multiple threat models can run in parallel against separate instances. Installation uses pip, and the guide assumes Python and asyncio familiarity. The package supports Python 3.11 through 3.13, not 3.14, and PyPI lists Alpha status under the MIT license. Persisted trajectories may contain sensitive test content.

Who it suits

SuperRed is aimed at red-teamers, system builders and evaluators who can use Python and asyncio. It suits teams evaluating staged, sandboxed or live AI systems with configurable threat models.

What it does well

35 attacker, target and benchmark modules
Parallel threat-model execution
Live dashboard and web reports
Configurable security scopes and budgets
Supports prompt injection and jailbreak tests

What to know first

Python 3.14 is unsupported
Guide assumes Python and asyncio knowledge
Persisted trajectories are not scrubbed
Alpha development status

Verdict

SuperRed provides modular, self-hosted testing for chatbot and agent security. Its configurable scopes, parallel runs and detailed reports support structured evaluations, but users must manage sensitive persisted trajectories and compatible Python versions.

Facts from SuperRed's own pages

Attacks and benchmarks: The module catalogue lists 35 modules: 21 attackers, 6 targets, and 8 benchmarks. · rdi.berkeley.edu, 4 Oct 2026
Composable components: It treats the attacker, system under test, and benchmark as separate interchangeable modules coordinated by a controller. · rdi.berkeley.edu, 4 Oct 2026
Composable modules: It keeps attackers, systems under test, and security claims as interchangeable modules coordinated by a controller. · rdi.berkeley.edu, 4 Oct 2026
Evaluation reports: Runs record model, cost, and success rate; the framework provides live progress and browsable reports with per-task details and trajectories. · rdi.berkeley.edu, 4 Oct 2026
Example modules: Listed modules include PAIR, TAP, AutoDAN-Turbo, Crescendo, AgentVigil, HarmBench, AgentDojo, and DecodingTrust-Agent. · rdi.berkeley.edu, 4 Oct 2026
Installation: The guide installs the framework with pip install superred and describes it as a Python framework whose guide assumes familiarity with Python and asyncio. · rdi.berkeley.edu, 4 Oct 2026
Integration: The getting-started example connects to any LiteLLM-compatible model endpoint using a base URL and API key. · rdi.berkeley.edu, 4 Oct 2026
Intended audience: The getting-started guide is for people who can read Python and have seen asyncio, and PyPI lists Science/Research as an intended audience. · rdi.berkeley.edu, 4 Oct 2026
Intended users: The project describes use by red-teamers, system builders, and evaluators, and its guide covers wrapping systems, writing attackers, defining success criteria, and running evaluations. · rdi.berkeley.edu, 4 Oct 2026
License and maturity: PyPI lists the package under the MIT license and classifies its development status as Alpha. · pypi.org, 4 Oct 2026
Maker: The site says SuperRed was made at the University of California, Berkeley. · rdi.berkeley.edu, 4 Oct 2026
Metrics and reports: Runs record model, cost, and success rate, with a live terminal dashboard and a web report for results. · rdi.berkeley.edu, 4 Oct 2026
Model endpoints: The example target uses a LiteLLM-compatible endpoint with a base URL and API key; the guide says most LLM-driven attackers also call models through LiteLLM. · rdi.berkeley.edu, 4 Oct 2026
Parallel runs: The framework can run multiple threat models in parallel, each against its own system instance. · rdi.berkeley.edu, 4 Oct 2026
Purpose: SuperRed is an open-source framework for red-teaming chatbots, agents, and assistants by testing whether attacks can violate security properties. · rdi.berkeley.edu, 4 Oct 2026
Requirements: The package requires Python 3.11 through 3.13; Python 3.14 is not supported. · pypi.org, 4 Oct 2026
Scale: The controller can run many threat models in parallel, each against its own system instance. · rdi.berkeley.edu, 4 Oct 2026
Security controls: The controller filters attacker-visible and injectable surfaces according to security-domain scopes and can cap attacker model spend per task. · rdi.berkeley.edu, 4 Oct 2026
Security scope: The controller filters which controllables, observables, trajectory entries, and evaluation sub-scores the optimizer can access according to the configured scope. · rdi.berkeley.edu, 4 Oct 2026
Sensitive results: Persisted trajectories are not scrubbed and may contain jailbreaks, planted secrets, and exfiltrated content, so the maker says to treat the results folder as sensitive. · rdi.berkeley.edu, 4 Oct 2026
Supported targets: Targets can wrap fixed-response fixtures, simulated environments, sandboxes, or live deployments, and the guide recommends staging or throwaway instances for real systems. · rdi.berkeley.edu, 4 Oct 2026
Target types: Targets can wrap fixed-response fixtures, simulated environments, sandboxes, or live deployments. · rdi.berkeley.edu, 4 Oct 2026
Threat models: Each run can define attacker model, per-task budget, trust-boundary access, and whether benchmark feedback is visible to the attacker. · rdi.berkeley.edu, 4 Oct 2026

Where SuperRed stands on Freedom251

AI Security Testing Tools#24 of 29