Marvin Okafor built an agent to generate tests for surviving code mutations—and found that eight defects in his evaluation harness could make its results look better, cleaner, or more convincing than they were. His account is a useful case study in a difficult engineering truth: an agent’s score is only as trustworthy as the instrument that measures it.
Okafor describes a project-specific experiment, not a general benchmark. The agent tried to strengthen test suites by targeting surviving mutants, while the harness decided whether its generated tests deserved to be kept. When the harness itself was wrong, it risked turning a measurement into a misleading result.
As an Amazon Associate I earn from qualifying purchases.
The account comes from Okafor’s article on DEV Community. The repository was described as private, so the code and reported results have not been independently reproduced here.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWhat mutation testing measures—and what coverage does not
Mutation testing makes small, deliberate changes to code and runs the test suite against each changed version. A mutant is “killed” when tests fail; it “survives” when they do not. A surviving mutant is evidence that the tested suite did not detect that particular change. It is not, by itself, proof that the original program has a production defect.
Line coverage answers a different question: did a test execute this line? Execution does not establish that the test would notice an incorrect result. A test can pass through a line while making no assertion about the behavior that line controls. Mutation testing probes whether the suite responds to selected changes, making it a fault-detection signal rather than a substitute for judgment about test quality.
Okafor’s framing is direct: “If no test fails, that is a bug your suite cannot detect.” In context, the statement is about the tested suite’s ability to detect a change—not a claim that every surviving mutant corresponds to a real-world bug.
How the agent tried to improve weak tests
The agent identified a surviving mutant, supplied a model with the mutation diff, and asked it to create a test. A generated test was accepted only if it passed against clean code and failed against the targeted mutant. On a retry, the agent could provide actual pytest output. The acceptance decision came from the subprocess exit code, rather than asking the model to judge whether its own test worked.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsThat two-sided gate matters:
- Passes on clean code: the test does not simply fail against the unmodified program.
- Fails on the mutant: the test detects the specific change it was intended to catch.
A test that only satisfies the first condition may be valid but ineffective. One that fails on both clean and mutated code may be broken. Checking both outcomes separates a plausible test from one that actually detects its target mutation.
Eight defects that undermined the measurement
Okafor reports finding eight harness defects. He says each would have made results look better, cleaner, or more publishable. That direction-of-bias assessment is his account; without the private code, it cannot be independently audited.
- Mutations were written to a temporary copy, but imports came from the checkout. Editable installs for packages using a
srclayout resolved imports to the original code instead of the mutated copy. The mutations were therefore invisible, and three targets appeared to score 0.000. - Parallel execution made one target unstable. Running mutants concurrently produced three different survivor sets across four runs for a target using asynchronous I/O.
- The test-file picker selected the wrong file. On the hardest target, the harness ran a test file that was not the intended one.
- A batch-level classifier blurred individual test results. It classified batches rather than tests, allowing one strong test to make a whole batch of 69 appear strong.
- Test reconstruction dropped shared imports. Rebuilding a generated test without its shared imports manufactured failures that did not reflect the test’s intended behavior.
- The extractor missed valid test methods. It scanned only top-level functions, discarding valid
unittest.TestCaseresponses. The retry loop then received a harness error instead of pytest output. - An assertion was mistaken for no assertion. The classifier treated
self.assertEqual(...)as “no assertion,” potentially creating the very outcome the author had hypothesized. - A metric was applied where it was undefined. A pre-registered metric did not apply to dunder-dispatched code such as
__call__and__or__, yet appeared as a real, near-zero rate.
How Okafor checked whether the instrument was lying
Rather than relying only on code review, Okafor says he predicted what specific checks should report and then investigated when their outcomes were wrong. He summarized the approach this way: “None of them was found by reading code.” He says each defect surfaced when he anticipated a check’s result and found that the actual result contradicted the prediction.
That method makes the expected outcome part of the test. If a mutant is supposed to be imported from a temporary directory, check where the running process imports it from. If a classifier is meant to score tests individually, give it a batch where one test is strong and the rest are not. If a metric is supposed to apply only to certain dispatch patterns, verify that unsupported cases are marked as inapplicable rather than given a plausible-looking number.
Free tools Windows power users keep installed
One-click scans. No signup required.
Okafor’s key question for measurement design was: “What would my instrument look like if it were lying to me?” A practical extension is to write down not only the expected result, but also how a failure could bias the score. That makes it easier to spot errors that produce tidy, favorable numbers instead of obvious crashes.
What the reported numbers do—and do not—show
Okafor reports that the project examined 455 mutants across 12 Python libraries, with 133 surviving the existing test suites. He says 53 of the 133 survivors were on lines the tests had executed. After he widened per-target test commands by between 6 and 40 times, the reported count changed from 54 to 53, while previously unreachable mutations instead became kills. These are figures from his project, not independently validated measurements or industry benchmarks.
Rank #4
In a comparison covering 15 mutants, the single-test baseline reportedly killed 1, while the agent killed 9; the reported keep rate was 60%. The author says the run covered only two of ten targets before the API budget ran out, and those were the targets where the baseline performed worst. This was not a random or representative sample.
Other details further narrow what can be concluded:
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
- Okafor reports that all nine kills came from the two cheapest mutation types.
- He reports no cross-function transfer, and says seven kept tests killed only the mutation they targeted.
- He expected accepted tests might often be vacuous. Instead, he reports an empty “none” category, eight of nine kills as real assertion failures, and six discarded drafts that passed on clean code but did not detect the mutation. In this small run, the gate appears to have filtered valid but ineffective drafts, rather than merely rejecting broken tests.
These results suggest the agent could generate tests that detected targeted changes in some cases. They do not establish how it would perform across targets selected in advance, more mutation types, or broader testing conditions. The sample was small, selectively reached, and concentrated in two inexpensive mutation types.
Best Value
Reproducibility is part of the result
For a clean-clone check, Okafor says he ran each target three times serially and required byte-identical survivor sets. Eleven of the twelve targets reportedly reproduced the same sets; one varied, and that result was disclosed in the README. This is the author’s reported protocol and outcome, not an independent reproduction.
That check speaks to repeatability under those conditions, not to whether the chosen mutants or tests adequately represent software quality. The asynchronous-I/O target’s earlier variation under parallel execution also shows why the execution mode is not a minor implementation detail: changing it changed the observed survivor sets.
What to carry into another agent evaluation
Okafor’s experience supports a practical discipline for teams evaluating test-generation systems:
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →- Define the outcome before running the agent. Specify what counts as a valid test, a detected mutation, a discarded draft, and an inapplicable metric.
- Test the evaluation pipeline with controlled cases. Verify import paths, mutation visibility, test selection, reconstruction, extraction, assertion recognition, and metric handling separately.
- Predict outcomes in advance. For each check, record what should happen and how an incorrect result could bias the reported score.
- Compare clean and mutated code. Require a candidate test to pass on the clean program and fail on the targeted mutant; use the test runner’s result as the acceptance signal.
- Control and disclose execution conditions. If concurrency can affect results, compare serial and parallel runs and state which results are reproducible.
- Report the selection process and limits. State which targets and mutation types were covered, what was not reached, and whether targets were chosen through a representative plan.
The broader lesson is methodological, not a claim that this agent—or any other—has demonstrated general superiority. Evaluation infrastructure is part of the experiment. A convincing number can still be wrong if the harness imports the wrong code, runs the wrong tests, misreads assertions, or reports an undefined metric as a real score.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
