DriversRecommendedOutdated drivers can make a good PC feel brokenScan driver issues before chasing fixes manually.Scan NowHalloween DealsAmazon USHalloween week deals: useful picks before month-endAmazon US: current deals, useful picks and tech finds.Check DealsSlow PC?RecommendedPC slow today? Run a repair scan before it gets worseResolve common Windows issues and optimize system performance.Scan Now×
FREEDOM251

Freedom251 › Blog › mutation testing

My Mutation Score Was 1.000 in Two Places—and the Denominators Disagreed

By Freedom251 Team · · 3 min read

What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Two mutation-testing reports can both show 1.000 and still measure different things. The score is a ratio, so its denominator—and the mutant population and status rules behind it—are essential to interpreting the result. Before comparing the reports, find each formula and reconcile its raw counts.

What a mutation score of 1.000 tells you

A score of 1.000 means every mutant counted under that report’s rules was classified as killed or otherwise defeated. It does not, by itself, show that the reports counted the same mutants or that the test suite would detect every real-world defect.

As an Amazon Associate I earn from qualifying purchases.

The conventional mutation score is the number of killed mutants divided by the number of non-equivalent mutants. In that definition, the denominator can include mutants that tests did not cover. A covered-code score instead can divide defeated mutants by the mutants covered by tests, measuring performance on a narrower population. The conventional definition and a covered-code formula described in an academic thesis illustrate why the same displayed value need not represent the same measurement.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Why the denominators can differ

All non-equivalent mutants versus covered mutants

A report using all non-equivalent mutants asks whether the tests defeated the eligible mutants in the run, including those that may be uncovered. A covered-code score asks whether tests defeated the mutants that were covered. If uncovered mutants are excluded from the latter denominator, a 1.000 there does not establish that the broader population was fully defeated.

Equivalent-mutant decisions

Equivalent mutants behave the same as the original program and are excluded from the conventional denominator. Determining equivalence is difficult; the Luxembourg mutation-testing repository notes that the general problem cannot be solved automatically. Reports may therefore differ in which mutants they identify, exclude, or leave unresolved. Do not assume equivalent-mutant handling matches across tools or runs.

What counts as defeated

Denominator differences are not the only source of disagreement. In the thesis’s covered-code formula, defeated mutants include killed, timed-out, and error outcomes. That is a specific formula, not a universal rule for all tools. Inspect each report’s definitions for killed, defeated, timeout, error, skipped, and uncovered statuses before comparing its score.

Reconcile the reports using counts, not rounded scores

For each report, record the formula and the population behind it. Use the tool’s report or documentation for the specific version and run; the title alone does not identify either report’s rules.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
  1. Write down the formula. Identify whether the score is conventional, covered-code, or tool-specific, and what its numerator and denominator represent.
  2. Record run scope and generated mutants. Note whether the run covers a project, module, changed code, or selected operators, and capture the total generated mutants.
  3. Break down the outcomes. Record killed or defeated, surviving, uncovered, excluded, and equivalent mutants. Also capture timeout and error counts, and establish how each is treated in the score.
  4. Check selection and execution rules. Look for sampling, filtering, or incremental execution that might have changed which mutants entered the report.
  5. Recalculate from raw counts. Apply each report’s stated formula and compare the populations. If the denominators count different populations, report the scores as different measures—even if both display 1.000.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

What can—and cannot—be concluded from this case

The general explanation is clear: denominator conventions and status rules can make two 1.000 scores represent different measurements. But without the tools, versions, project scope, raw counts, and report excerpts, it is not possible to determine which specific report is labeled correctly or reproduce either denominator. To settle the discrepancy, obtain those details and check the documentation for each tool version.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

mutation testingsoftware qualitysoftware testingTest Coverage

Leave a Reply

Your email address will not be published. Required fields are marked *

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Recommended PC Tool
Recommended PC Tool
Windows Errors? Fix Them Before They SpreadFree repair scan
Outdated Drivers Are Slowing You DownFree scan - exact matches

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.