Free tools Windows power users keep installed
One-click scans. No signup required.
A flaky test that fails and then passes on rerun has not been fixed. The green retry is a new observation, not proof that the test or the code is reliable. Use it as a clue, investigate the conditions behind the failure, and restore the test’s value as a regression signal.
What a flaky test tells you—and what it does not
Martin Fowler defines a non-deterministic test as one that “passes sometimes and fails sometimes, without any noticeable change in the code, tests, or environment.” (Martin Fowler, “Eradicating Non-Determinism in Tests,” first published April 14, 2011.) A failure followed by a pass is consistent with that pattern, but one retry alone does not establish the cause.
As an Amazon Associate I earn from qualifying purchases.
The practical problem is lost signal: when a regression test fails intermittently, it is harder to tell whether the code has a defect or the test is responding to variable conditions. Retrying until green can hide that uncertainty without explaining it. If a team grows accustomed to dismissing such failures, confidence in other test results can weaken as well.
Recommended Free Tools
Investigate the conditions that can change the result
Fowler identifies recurring sources of non-determinism. Treat these as leads to check, not as proof of what happened in a particular test:
- Shared state or weak isolation: one test may leave data or process-wide state that affects another.
- Asynchronous work checked with a fixed sleep: a guessed delay may be too short on a loaded machine and unnecessarily long when work finishes quickly.
- Remote-service dependencies: network or service behavior can vary beyond the test’s control.
- Direct use of the system clock: time-dependent behavior may change with the clock or with when the test runs.
- Resource leaks: open resources or incomplete cleanup can affect later tests, especially in longer runs.
A repair-oriented workflow
- Record and reproduce the failure. Capture the failing test, relevant logs, and the conditions around the run. Compare code changes with variation in state, timing, machine load, and external dependencies. A successful rerun is useful data, but it does not replace this investigation.
- Control the starting state and isolate tests. Make setup predictable, and ensure a test cannot leave data or process-wide state that changes another test’s result.
- Wait for observable asynchronous behavior. Replace a fixed sleep with a wait for the condition the test actually needs, bounded by a timeout. This avoids relying on a guessed interval while preventing an indefinite wait.
- Make unstable dependencies controllable where appropriate. A test double can keep a remote service from making a test’s result depend on the network; validate the double’s contract separately. Inject a controllable clock for behavior that depends on time.
- Inspect teardown and resource cleanup. Look for resources that remain open or state that persists, particularly when failures appear after a long run or seem to move between tests.
- Contain only when necessary, and assign repair ownership. Fowler advises: “Place any non-deterministic test in a quarantined area. (But fix quarantined tests quickly.)” Quarantine can keep an unreliable test from disrupting immediate feedback, but it should have an owner and a prompt repair expectation rather than becoming a permanent destination.
Retry, quarantine, or fix?
These actions serve different purposes. A retry gathers another result; quarantine contains disruption; a repair addresses the source of the unreliable signal. Fowler’s guidance supports quarantine as temporary containment and prompt repair, but does not prescribe a retry count or a particular CI setting.
| Action | Effect on the regression signal | Root cause | Feedback and ownership |
|---|---|---|---|
| Retry | A pass on rerun does not restore trust in the test; it is another observation. | Does not, by itself, expose or fix the cause. | Adds another run before a result; no repair owner or deadline is implied. |
| Quarantine | Contains disruption from the test, but the test no longer provides normal regression coverage while quarantined. | Does not fix the cause. | Use temporarily, with an owner and prompt repair expectation. |
| Fix | Can restore a trustworthy regression signal once the cause is addressed and the test behaves reliably. | Targets the cause rather than masking it. | Requires investigation and repair; the source does not specify a duration. |
What the guidance does—and does not—establish
Fowler’s article is engineering guidance, first published in 2011, not a new empirical study. It supplies no prevalence statistic, so there is no grounded percentage to attach to how often tests are flaky. It also does not establish what caused any particular failure or prescribe a universal retry policy. For further reading on test design, Fowler recommends Gerard Meszaros’s xUnit Test Patterns.
Quick Recap
Best Value
Rank #4
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →
