Freedom251 › Mutation Testing Tools
Cosmic Ray
What Cosmic Ray is
Cosmic Ray is a free mutation-testing tool for Python 3. It changes source code in small ways, then runs a test suite for each change to check whether tests detect altered behavior. This gives Python teams a way to examine whether tests verify behavior, beyond knowing whether a line was executed. Cosmic Ray supports unittest, pytest, and nose. A TOML configuration defines modules, test command, timeout, exclusions, and distributor. Its local distributor runs tests serially; the HTTP distributor sends work to local or remote workers for parallel runs, with a separate code copy needed for each worker. Reports are available through cr-report and cr-html, and users can add distributor and mutation-operator plugins. Included filters can skip mutations by operator, comment, or Git branch changes. The tool applies mutations directly to code on disk, so the documentation advises committing changes first. Runs may take a long time, and full configuration options are not currently well documented. Tests sharing a module with production code may also be mutated unless separated or excluded.
Who it suits
Cosmic Ray suits Python project teams who want to check whether their tests catch changes in behavior. It may be useful when code coverage alone does not answer that question.
What it does well
- Free to use under the MIT license.
- Supports unittest, pytest, and nose.
- HTTP distribution can run work across local or remote workers.
- Reports are available in summary and HTML formats.
- Mutation filters can skip selected changes.
What to know first
- Mutation runs can take a long time.
- Mutations are applied directly to code on disk.
- Configuration options are not fully documented.
- Tests sharing production modules may be mutated.
Getting started
- Install Cosmic Ray with `pip install cosmic-ray`.
- Create a TOML configuration for modules, test command, timeout, exclusions and distributor.
- Select the local distributor for serial runs or the HTTP distributor for worker-based parallel runs.
- Use cr-report for a session summary or cr-html for an HTML mutation report.
Verdict
Cosmic Ray provides a configurable way to assess Python tests through mutation runs, with serial and distributed execution options. Teams should account for potentially long runs and protect their working code before execution.
Questions
How much does Cosmic Ray cost?
Cosmic Ray is free, and its repository identifies it as MIT licensed.
Which Python test frameworks does it support?
The documentation shows unittest and describes using pytest or nose.
Can Cosmic Ray run tests in parallel?
Yes. Its HTTP distributor sends work to local or remote workers. Each worker needs its own copy of the code under test.
What platforms does it support?
The listed platforms are Linux and self-hosted.
Who is credited as the maker?
The project documentation credits Sixty North AS.
What reports does it provide?
It includes cr-report for session summaries and cr-html for HTML reports showing the mutations used.
Facts from Cosmic Ray's own pages
- Code handling
- Cosmic Ray applies mutations directly to code on disk, and the documentation advises committing changes first because a mutation might not be correctly undone after an exception or crash. · cosmic-ray.readthedocs.io, 3 Oct 2026
- Configuration limitation
- The documentation says the full set of configuration options is not currently well documented. · cosmic-ray.readthedocs.io, 3 Oct 2026
- Custom extensions
- Users can provide their own distributor and mutation operator plugins. · cosmic-ray.readthedocs.io, 3 Oct 2026
- Execution modes
- The local distributor runs tests serially on the local machine, and the HTTP distributor sends work to local or remote workers that can run tests in parallel. · cosmic-ray.readthedocs.io, 4 Oct 2026
- Extensibility
- Users can extend Cosmic Ray with their own mutation operators exposed through plugins. · cosmic-ray.readthedocs.io, 4 Oct 2026
- Install
- The tutorial says to install Cosmic Ray with `pip install cosmic-ray`. · cosmic-ray.readthedocs.io, 4 Oct 2026
- Intended users
- The documentation presents Cosmic Ray for Python projects whose teams want to check whether their test suites meaningfully verify code behavior. · cosmic-ray.readthedocs.io, 3 Oct 2026
- License
- The project repository identifies the software as MIT licensed. · github.com, 4 Oct 2026
- Maker
- The project documentation credits Sixty North AS. · cosmic-ray.readthedocs.io, 4 Oct 2026
- Mutation filters
- Included filters can skip mutations by operator name, a `# pragma: no mutate` comment, or whether the mutated line changed relative to a Git branch. · cosmic-ray.readthedocs.io, 4 Oct 2026
- Notable limitation
- Cosmic Ray may mutate test code if it shares a module with the production code under test; the documentation recommends keeping tests separate or excluding their modules. · cosmic-ray.readthedocs.io, 4 Oct 2026
- Parallel execution
- The HTTP distributor sends mutation and test requests to workers that can run locally or remotely; each worker needs its own copy of the code under test. · cosmic-ray.readthedocs.io, 3 Oct 2026
- Purpose
- Cosmic Ray is a mutation testing tool for Python 3 that makes small changes to production code and runs the test suite against each change. · cosmic-ray.readthedocs.io, 4 Oct 2026
- Reports
- Cosmic Ray provides `cr-report` for session summaries and `cr-html` for generating an HTML report that shows the mutation used. · cosmic-ray.readthedocs.io, 4 Oct 2026
- Runtime consideration
- Mutation testing can take a long time because the tool runs the test suite for each mutation; the documentation gives an example where 1,000 mutations and a 10-second test suite take about 2.7 hours. · cosmic-ray.readthedocs.io, 3 Oct 2026
- Runtime limit
- A configured timeout ends a mutant test that runs too long, and Cosmic Ray marks that mutant incompetent. · cosmic-ray.readthedocs.io, 4 Oct 2026
- Target users
- The documentation describes Cosmic Ray as a tool for Python projects and says it has been used on projects ranging from assemblers to oil exploration software. · cosmic-ray.readthedocs.io, 4 Oct 2026
- Test commands
- A TOML configuration specifies the modules to mutate, test command, timeout, exclusions, and distributor. · cosmic-ray.readthedocs.io, 3 Oct 2026
- Testing value
- Mutation testing checks whether tests detect changes in behavior, while code coverage indicates whether a line was executed. · cosmic-ray.readthedocs.io, 4 Oct 2026
- Test runners
- The documentation shows test commands using `unittest` and says `pytest` and `nose` can stop after a failing test with the `-x` option. · cosmic-ray.readthedocs.io, 4 Oct 2026
Where Cosmic Ray stands on Freedom251
- Mutation Testing Tools#8 of 25