Mutatest

Mutatest is a command-line mutation-testing tool for Python projects. It is best suited to developers who want targeted diagnostics and can work within its Python and test-runner constraints. Its coverage filtering and configurable thresholds are useful, but repeated test runs make whole-codebase checks on every commit a poor fit.

Overview

Mutatest checks whether tests catch small changes by generating valid mutations from Python’s Abstract Syntax Tree grammar. It changes local __pycache__ files rather than source files, keeping mutation runs from directly altering the source files themselves.

When coverage data is available, targets can be restricted to covered lines. Mutatest also samples mutation locations and possible changes randomly; a seed makes a trial reproducible. The trade-off is runtime: repeated suite runs make this a diagnostic tool, not a practical full-codebase check on every commit.

Key features

Configurable mutation runs

Command-line controls include run modes, mutation categories, file exclusions, timeouts, output reports, and thresholds for surviving mutants. Operators cover augmented assignment, binary and bitwise operations, Boolean logic, comparisons, conditionals, indexing, names, and slices. An API offers custom mutation controls for workflows that need more than the command-line settings.

Coverage and automated checks

Mutatest can use coverage data to focus mutations on lines tests reach, while a random seed supports repeatable sampling. It can also raise an exception when surviving mutants meet a configured threshold, allowing teams to use mutation results as a CI quality gate. These controls support targeted automation, though the cost of rerunning tests argues for selective use.

Pricing

Mutatest is free and open source under the MIT License. The Mutatest plan costs nothing, with no paid plan or trial to weigh against its capabilities.

Platforms

Mutatest supports Linux, macOS, and Windows, and the project says it has been tested on all three using Azure pipelines. It supports Python 3.7 and 3.8; multiprocessing is available on 3.8 but not 3.7. That narrow version range is a material constraint for teams using newer Python releases.

Although users can supply a test command, only pytest has been tested. Teams relying on another runner should treat compatibility as uncertain rather than assume it will work the same way.

Who it's for

Mutatest suits Python developers who want to investigate whether tests detect small code changes, especially when coverage data can focus the run and thresholds can flag survivors in automation. It is less appropriate for teams that need broad mutation checks on every commit, depend on parallel execution under Python 3.7, or require a tested runner other than pytest.

Pros and cons

Pros

• Focused runs: Coverage filtering can target lines tests reach, reducing wasted mutation checks.
• Useful controls: Sampling seeds, exclusions, timeouts, reports, and survivor thresholds support repeatable, tailored diagnostics.
• CI thresholding: Raising an exception at a survivor threshold makes mutation results usable as an automated gate.
• No license cost: The MIT-licensed project is free to use.

Cons

• Limited Python versions: Installation requires Python 3.7 or 3.8, restricting teams on other versions.
• Runner uncertainty: Only pytest has been tested, despite the ability to supply a test command.
• Slow for frequent broad runs: Repeated suite execution makes whole-codebase checks on every commit impractical.
• No multiprocessing on Python 3.7: Parallel execution is limited to Python 3.8.

Alternatives

For a broader comparison, browse Mutation Testing Tools. Consider ArcMutate if you want a freemium option with a free trial and Linux, macOS, self-hosted, and Windows platforms. Mutant is another freemium option, with a free plan for public repositories and a commercial monthly plan billed per developer.

pymute is a free Linux option that requires Rust and Cargo. pytest-gremlins is a free MIT-licensed Python package requiring Python 3.11 or later, which may suit projects on that version. Stryker Mutator is a free open-source option for JavaScript, TypeScript, C#, and Scala. Gambit is a free open-source project available on Linux, macOS, and Windows. Infection is a free option available on Linux, macOS, and the web. mutmut is a free option available on Linux, macOS, self-hosted environments, and Windows.

Verdict

Choose Mutatest if you need free, configurable mutation diagnostics for a Python project that fits its Python 3.7 or 3.8 requirement and pytest-tested workflow. Coverage targeting and CI thresholds give it practical value for focused checks. Look elsewhere if you need a tested alternative runner, broader Python-version support, or fast mutation checks across the whole codebase on every commit.