In an example reported by software engineer Marvin Okafor, an AI model generated 69 tests for a Python module. Every test passed on the original code, yet none detected 11 bugs Okafor had deliberately planted. The contrast exposes a basic testing distinction: a test can run successfully without checking whether the code behaves correctly.
Okafor then tested that distinction more systematically with mutation testing across selected Python-library modules. His results favor tests generated against specific mutations and kept only when they pass on clean code and fail on the targeted faulty version—but they are findings about one experiment, not a general ranking of AI coding systems.
As an Amazon Associate I earn from qualifying purchases.
How can 69 passing tests catch none of the bugs?
A test suite can pass because it confirms behavior that remains unchanged even when a bug is present. A test may call a function and check only that it returns something, for example, without checking the value that distinguishes correct behavior from an incorrect one. Passing tells you the test’s expectations were met on the code it ran; it does not, by itself, show that the test would notice a fault.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallThat was the point of Okafor’s opening example: the generated tests passed against the original Python module, but none failed when he introduced any of the 11 planted bugs. The example is separate from the larger experiment below; it should not be treated as a result drawn from the twelve-library comparison.
What did the larger experiment measure?
Mutation testing creates small, deliberate changes to code and checks whether a test suite fails. Okafor’s harness used edits such as flipping a comparison, changing a constant, or removing a raise. If a change survived the tests, that was evidence that the suite did not detect that particular mutation.
For AI-generated tests, Okafor retained a test only when it passed against clean code and failed against the specific mutation it was meant to catch. The outcome came from a subprocess exit code, rather than a model being asked to judge whether the test detected a fault. That gives the experiment a concrete pass/fail rule, though its meaning still depends on the mutations, targets, and harness used.
The killcheck repository describes the scope more narrowly: it tests whether a mutation hint, an execution gate, and a one-test-per-call setup improve results over comparison conditions for reachable survivors in selected modules. It explicitly says the experiment is not a general measure of whether agents write good tests.
What were the results across the selected Python targets?
Okafor reports generating 455 mutations across 12 Python-library targets. Of those, 133 survived the existing test suites. But only 53 of those 133 were on lines the suites actually executed; the other survivors were in code the tests did not reach.
Among the 53 reachable surviving mutations, Okafor reported these results:
| Approach | What the model did | Reported catches |
|---|---|---|
| Targeted generation with a pass/fail gate | Received a specific mutation hint; a generated test had to pass on clean code and fail on that mutation | 44 of 53 reachable survivors |
| One broad prompt | Received a general request to write more tests, without the same per-mutation targeting | 9 of 53 reachable survivors |
| Untargeted one-test-per-call | Was prompted to write one test per call without a specific mutation target | 2 of 53 reachable survivors |
According to the article, the approaches used the same model and token ceiling. These counts are the author’s findings from this setup, not independently replicated rates or evidence that targeted generation will outperform other methods in every codebase.
Rank #4
Why does reachability change the interpretation?
Line coverage records whether execution reached a line. Mutation testing asks a different question: if that line’s behavior were changed in a particular way, would a test fail? A line can be executed without its important behavior being asserted. Conversely, if tests never reach a line, stronger assertions on already-executed lines will not expose faults in that unvisited code.
Free tools Windows power users keep installed
One-click scans. No signup required.
In this experiment, 53 of the 133 surviving mutations were on executed lines. Okafor says that widening the test commands by six to 40 times changed the reachable-survivor count from 54 to 53. He interpreted the results as pointing to unreached code as the larger gap in these targets—not merely weak assertions on lines already covered. That interpretation applies to this target set and measurement, not to all software projects.
Best Value
Did the targeted tests generalize to other mutations?
The 44 retained targeted tests were highly specific in the initial accounting: Okafor reports zero cross-function transfer, and 36 tests caught exactly one mutation. Zero cross-function transfer does not mean that a test could not detect another mutation within the same function.
A later update to the repository adds an important qualification. When the frozen set of 44 tests was run against fresh reachable mutants, it caught 34 of 53. Okafor says transfer was observed within functions, but not across functions. The fresh population pooled to 92, below a preregistered minimum of 100, and two targets contributed 30 of the 53 reachable mutants. This is evidence of some within-function transfer in that holdout attempt, but the update’s own limits counsel against treating it as a broad generalization result.
What did the author find wrong with the harness?
Okafor reports finding 11 bugs in the harness itself, followed by three additional issues raised by readers after publication. He says each instrument problem either made results look better or made absence look like evidence. Examples included editable installs that hid mutations, parallel execution corrupting a target, a classifier using the wrong unit, stale bytecode, and a pytest outcome bucket matching a string the installed version did not emit.
According to Okafor, these problems were not found simply by reading the code: checks with predicted outcomes exposed them, and readers later found more by examining those checks. That history is a reason to publish and scrutinize evaluation harnesses, not proof that every evaluation is biased or unreliable.
What should developers take from the experiment?
- Passing and detecting faults are different properties. A suite can pass on a clean version while failing to detect a meaningful change in behavior.
- Coverage is not a substitute for checking outcomes. Coverage can reveal which code ran; mutation testing can probe whether tests react to selected changes in that code.
- Targeted generation had the strongest result in this setup. The advantage was measured against reachable survivors in selected modules, with a particular model, token ceiling, mutation set, and pass/fail gate.
- Unexecuted code remains a separate problem. Better assertions cannot catch faults in paths tests never exercise.
- Evaluation machinery needs its own checks. Predicted-outcome tests and external scrutiny helped uncover defects in this harness.
For a reader who wants to inspect the method, the repository’s quickstart is the practical resource. The author’s broader lesson is that an evaluation harness is worth publishing so its assumptions and failure modes can be examined.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
