Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →A green automated test run means the checks that ran met their encoded expectations. It does not prove that every important behavior was tested, that the expected results were correct, or that the checks observed the kind of outcome that matters. Here are four common ways a real defect can survive a passing suite—and how to investigate each one.
What a passing test run does—and does not—tell you
A passing run is evidence about particular checks, inputs, assertions, and the environment in which they ran. It does not establish that the application is defect-free. ISTQB’s testing principles explain this limit: testing can reveal defects, but cannot prove their absence (ISTQB testing principles).
As an Amazon Associate I earn from qualifying purchases.
When a user reports a bug after a green build, trace the chain: the requirement or user expectation, the behavior under test, the system boundary the test exercised, and the assertion that determined success. A gap at any point can let a defect through.
1. The broken path was never tested
A suite can repeatedly pass while omitting the user journey or edge case where the defect occurs. The tests may cover the ordinary checkout path, for example, but not a particular payment method, account state, or recovery step. Nothing in a green result says what scenarios were absent.
How to check for the gap
- Reproduce the defect and write down the exact steps, inputs, and relevant account or system state.
- Find the nearest existing test and compare its path and inputs with the failing case.
- Add a focused regression test that reaches the affected behavior and asserts the intended outcome.
- Keep the test alongside broader journey or edge-case checks; one new scenario does not guarantee that all related cases are covered.
AxonBuild describes audited examples where the relevant path lacked a working test. That is an illustration of a coverage gap, not evidence that more tests alone guarantee correctness (AxonBuild’s account).
2. The test expected the wrong result
A test can pass because its expected result is wrong. The assertion may encode a mistaken requirement, or it may simply bless the implementation’s existing behavior. In either case, the test confirms the bug rather than exposing it.
AxonBuild illustrates this with a generated test that asserted division by zero should return zero. That example is specific to its article; the broader failure mode is an oracle—the rule that decides what result is correct—that was never independently checked.
How to check the test oracle
- Compare the expected value with the product requirement, domain rule, or agreed user behavior—not just the current output.
- Ask who or what established the expected result and whether that source is authoritative for the behavior.
- Review boundary cases and error behavior, where assumptions are easy to encode incorrectly.
- When a test is derived from implementation details, validate its expectation independently before treating it as a correctness check.
3. A mock or stub bypassed the faulty production behavior
Test doubles are useful for isolating components, but they can hide defects if they replace the very behavior that needs verification. A test might confirm that a mocked payment service received a request while never exercising the production code that creates or records a sale.
AxonBuild reports a checkout suite that did not call the code responsible for creating a sale. That specific case is reported by AxonBuild; the general lesson is to verify the system boundary each test actually reaches.
How to check what the test really exercised
- Follow the call path from the test entry point to the behavior implicated by the bug.
- Identify every mock, stub, fake, or intercepted request on that path and what real code it replaces.
- Keep isolated unit tests where they are useful, but add an integration or end-to-end check when the defect depends on interactions across components.
- Assert the meaningful outcome—such as a persisted sale—not only that a substitute received a call.
4. The suite tested function, not appearance
An interface can remain technically operable while its visual presentation is broken. A registration dialog may submit successfully even if buttons overlap, controls are misplaced, or text is obscured. Functional tests do not automatically verify layout or visual correctness.
Rank #4
Qt describes tests passing despite broken dialog layouts because the checks validated function rather than appearance (Qt on balancing feature and visual testing).
Match the check to the requirement
- For a functional requirement, assert the action and result that matter.
- For a visual requirement, use an appropriate visual assertion, such as a reviewed screenshot comparison, or conduct a visual review.
- For usability, performance, or accessibility requirements, add checks designed to observe those outcomes; a functional pass does not cover them by implication.
Coverage and pass rate are not proof of correctness
Coverage describes which code or scenarios a measurement says were exercised; pass rate describes outcomes among the checks that ran. Neither alone establishes that the suite covered the important user behavior, used valid expectations, reached production dependencies, or observed visual and other non-functional requirements.
Best Value
For each escaped defect, ask four questions: Was the requirement and user path covered? Did the test reach the relevant real code and dependencies? Was the expected result specified independently? Did the assertion observe the kind of outcome that failed? Add a focused regression check where it can represent the requirement, and retain exploratory or visual review where it cannot.
Or skip the browser setup
If a visual regression check needs captured browser screenshots, ScreenshotNeo is a website screenshot API and MCP server for developers. A one-call capture of a page looks like this (see the ScreenshotNeo API documentation):
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpScreenshotNeo accepts cookie or consent banners like a visitor and removes more than 60 known consent platforms, newsletter popups, and chat widgets before capture; each step can be turned off. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits cost nothing, and response headers indicate the page verdict and billing status. Its MCP server gives AI agents tools for screenshots, page information, and PDF capture. The free plan includes 1,000 screenshots a month with no card; paid plans start at $5 for 3,000.
Sign up for ScreenshotNeo’s free plan.
Frequently Asked Questions
Why do my tests pass but the app doesn’t work?
A passing run only confirms the checks that ran met their assertions; the defect may be in an untested path, a wrong expectation, bypassed production behavior, or an outcome the suite does not observe.
Does 100% test coverage mean there are no bugs?
No. Coverage does not establish that assertions express the right behavior or that all relevant functional and non-functional requirements are verified.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
