What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
A passing test does not prove the code is correct if the test’s expected result was copied from the code’s behavior instead of checked against the requirement. That is the danger Gil Zilberfeld describes in his article: generated tests can reproduce a bug, then make it look verified.
How can a passing test still be wrong?
A test can be internally consistent and still assert the wrong outcome. If a test generator examines existing code and derives its expected result from what that code currently does, the test may confirm the implementation rather than check whether it meets the requirement. Zilberfeld calls this a “tautological test.”
As an Amazon Associate I earn from qualifying purchases.
The result is a green test run that offers false confidence: the test passes, but the behavior it approves may violate the rule the software is supposed to follow.
Recommended Free Tools
What does the expiration-date example show?
Zilberfeld illustrates the problem with a function that checks a credit card’s expiration date. In his example, the function compares today with the first day of the expiration month. That can cause it to report a card as expired before the month has ended.
A generated test for a card expiring in the current month expects the function to return True in the middle of that month. But under the requirement described in the article, the card remains valid until the end of its expiration month, so the expected result should be False. The test is plausible and may pass against the function, yet it encodes the wrong rule.
This is an illustrative example, not evidence about how often generated tests make this mistake. Zilberfeld’s article does not quantify the failure rate or compare generated tests with human-written ones.
Why did it generate the wrong test?
The central risk is that generating a test from an implementation can preserve the implementation’s assumptions—including its defects. A test generator can produce a well-formed test without knowing whether the expected outcome matches the product requirement. If nobody checks that expectation independently, a test suite can accumulate passing checks that validate the wrong behavior.
For that reason, a green result answers a limited question: did the code produce the result this test expected? It does not, by itself, answer whether that result is correct.
How should teams review generated tests?
Zilberfeld recommends treating the origin and review of tests as part of quality control, not assuming that a larger test count means greater confidence.
- Identify generated tests. Know which tests were generated and which are trusted, rather than treating every passing test as equally authoritative.
- Make human review visible. Ask what code and tests received human review, and check that the expected outcomes follow the requirement.
- Focus on risky logic. Exercise high-risk behavior extensively, especially boundaries and edge cases where an incorrect assumption can change the result.
- Look for tests that pass when they should fail. Compare expected outcomes with the stated requirement; correct tests whose assertions endorse an invalid result.
- Teach the practice. Developers need to learn to distinguish “the implementation does this” from “the requirement says it should do this.”
What does a green test run actually establish?
It establishes that the tested code matched the test’s assertion under the conditions the test exercised. Confidence in correctness requires another check: whether that assertion reflects the intended behavior, including relevant edge cases. As Zilberfeld puts it, “The irony is that while we finally got more tests, our confidence in them is lower.”
Rank #4
His point is not that generated tests are inherently useless. It is that test volume cannot substitute for reviewing the requirement and the expected result. A test becomes useful evidence only when its assertion is worth trusting.
Free tools Windows power users keep installed
One-click scans. No signup required.
Quick Recap
Best Value
Read Gil Zilberfeld’s article on DEV Community.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
