A passing test is trustworthy only if it exercises the behavior its name promises and checks an outcome that a plausible bug could change. GitLab puts the test directly: “A test that cannot fail is not providing coverage.” The open-gsd project’s testing standards offer concrete examples of how tests can go green without proving what they claim; the same design mistakes can happen in hand-written or generated tests.
What it means for a test to pass for the wrong reason
A test can pass while failing to protect the behavior implied by its name. That happens when its setup does not create the stated scenario, its action calls the wrong code path, or its assertion accepts both correct and defective outcomes.
As an Amazon Associate I earn from qualifying purchases.
These are sometimes called vacuous tests or pass-always tests. They are different from flaky tests: a vacuous test can pass consistently without checking the intended behavior, while a flaky test’s result changes with execution conditions such as test order or shared state.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Examples from the open-gsd testing standards
The open-gsd project’s testing standards require a test to exercise the behavior it names and to make an assertion that could fail if a plausible defect were introduced. They identify assert(true) and checking a value that is always set as non-compliant: neither distinguishes working behavior from broken behavior.
A timeout test that checks too little
The standards describe a test labeled as handling a timeout that only checked that the call did not throw and that effectiveRoot was a string. That assertion could pass even if timeout handling selected the wrong fallback. The corrected example checks a specific fallback object, including its effective root, mode, and reason.
A test name that does not match its action
A name is a promise about the scenario and behavior being tested, not evidence that the test actually exercises them. GitLab warns that a copied assertion may call the wrong method and still pass, and that setup must match the scenario named by the test. A test called “uses fallback on timeout,” for example, needs setup that actually causes the timeout path and an action that reaches the relevant code.
How to inspect a green test
Review the test as a short chain: scenario, action, observable result. Each link should match the behavior the test claims to protect.
- Scenario: Does the setup create the condition in the test name, such as a timeout, missing record, or invalid input?
- Action: Does the test invoke the function or path it claims to cover, rather than a nearby or copied method?
- Assertion: Is the expected outcome specific enough to distinguish correct behavior from a plausible defect?
- Failure check: Would removing or changing the behavior make this test fail for the expected reason? GitLab recommends inverting the condition or removing the behavior to check that a test’s failure is meaningful.
- Mock boundary: Does a mock replace an external dependency, or has it replaced the very behavior the test claims to verify? The open-gsd standards caution against mocking the system under test itself.
- Error and fallback paths: Does the test check the particular error, fallback, or degraded result, rather than merely confirming that the call returned or did not throw?
GitLab’s testing best practices also emphasize choosing assertions that distinguish nearby cases. A broad type check can establish that a value is a string; it cannot establish that the string is the correct fallback.
Coverage is not the same as fault detection
Code coverage records which code execution reached. By itself, it does not show whether a test would detect a regression. The 2016 paper “Will My Tests Tell Me If I Break This Code?” examined Java open-source projects using mutation testing. Its authors concluded that coverage was an effectiveness indicator for unit tests in that study, but not for system tests. That is a study-specific finding, not a universal rule about every project or test suite.
What the different checks tell you
| Approach | What it checks | What it cannot establish alone |
|---|---|---|
| Code coverage | Whether execution reached code. | Whether tests detect faults in that code. |
| Static checks, such as Vacuous | Whether source patterns suggest missing meaningful assertions or swallowed failures. | Whether every flagged test is defective or every real defect is detected; pattern-based checks can have exceptions, false positives, and blind spots. |
| Mutation testing | Whether tests catch deliberate changes to the code. | Whether the chosen mutations represent every meaningful defect; Vacuous describes this as more thorough analysis with higher runtime cost. |
| Review and targeted test inversion | Whether a named test fails when its behavior is broken, and whether it fails for the intended reason. | Whether the whole suite is free of unrelated gaps or state-dependent failures. |
Vacuous describes its static checks as complementary to mutation testing, not a replacement for it. Its documentation says the maintainers checked approximately 29,000 tests across named open-source suites and manually read each finding; they report that “Roughly 2% of tests in those projects can’t fail.” That is a project-reported result for those suites, not an estimate for tests generally.
Rank #4
When test order and shared state undermine confidence
A test can be meaningful in isolation yet unreliable in a suite if it depends on state left by another test, a particular dataset, or a specific execution order. GitLab’s unhealthy tests guidance describes state leakage and assumptions about datasets or order, including hard-coded identifiers presumed not to exist. Its best-practices guide recommends helpers that create non-existing records rather than arbitrary IDs, and notes that new spec files run in randomized order.
Free tools Windows power users keep installed
One-click scans. No signup required.
This is a separate failure mode from a vacuous assertion. An order-dependent test may fail or pass depending on context; a pass-always test can stay green because its assertion does not discriminate between working and broken behavior. Both reduce confidence, but they call for different fixes: make assertions meaningful, and make tests independent of hidden suite state.
Best Value
A practical repair sequence
- Rewrite the test name around the behavior. State the condition and expected result, such as the fallback selected when a timeout occurs.
- Make setup produce that condition. Avoid relying on an assumption about existing data or earlier tests.
- Call the relevant behavior. Check for copied or mismatched method calls, especially when the assertion still passes.
- Assert the distinguishing result. Check the specific fallback fields, state change, or side effect that would differ if the behavior were wrong.
- Challenge the test. Temporarily invert the condition or remove the behavior and confirm that the test fails for the reason its name describes.
- Use deeper checks where warranted. Static analysis can flag suspicious patterns; mutation testing can probe whether deliberate code changes are caught, at additional runtime cost.
For its own standards, open-gsd says code review is the primary enforcement for some of these properties and acknowledges that pattern scans can produce false positives. That is the policy of this one project, not a claim about how other projects enforce test quality.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
