To improve test stability, first confirm that a failure is intermittent, then make the test’s starting state, timing, dependencies, and execution environment more controlled. A retry that passes is evidence of flakiness—not proof that the original failure was harmless. Keep retries or quarantine visible and temporary while you find and fix the cause.
What makes a test flaky?
A flaky, or nondeterministic, test sometimes passes and sometimes fails despite effectively unchanged code and inputs. That weakens the value of the test suite’s signal: a failure may be a real regression, but intermittent failures make it harder to tell. Martin Fowler describes nondeterminism as a test that passes and fails without a noticeable change in code, tests, or environment (“Eradicating Non-Determinism in Tests”).
As an Amazon Associate I earn from qualifying purchases.
Typical sources are uncontrolled state or execution conditions: shared data, incomplete setup or cleanup, order dependence, wall-clock time, asynchronous races, remote services, and insufficient runner resources. Treat a flaky result as a symptom to diagnose rather than a test failure to dismiss.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsHow to diagnose a flaky test
1. Confirm and record the failure
Capture the code revision, test name, environment, failure output, and relevant logs. Run the suspect test independently and note whether it passes or fails. A pass on retry establishes neither correctness nor harmlessness; it only supports the possibility that the result is intermittent. Google’s guidance recommends investigating rather than treating a retry as a fix (Google Testing Blog, 2021).
2. Check order dependence and state
Look for fixtures, singleton or static state, database rows, files, caches, and other data that one test can leave behind for another. Verify that initialization is complete before assertions run and that cleanup occurs even when a test fails. Run the test in different orders, or alone, to see whether another test or shared fixture changes its outcome.
3. Inspect time and asynchronous operations
Check whether the test depends on the wall clock, crosses a time boundary, or assumes fixture timestamps that do not match the current time. For asynchronous work, identify the exact application state that must be reached and wait for that state with a timeout. Include useful context in the timeout failure so the next run explains what was still missing.
A fixed sleep is not a durable synchronization strategy: the operation may take longer than the delay, while a longer delay wastes time on every run. Google explicitly cautions against arbitrary delays because they can become flaky again and slow tests unnecessarily (Google Testing Blog, 2021).
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →4. Examine external dependencies and the runner
Check whether a remote service, third-party API, or variable network response affects the test. Inspect runner logs and resource use as well: a system under test may fail when it does not receive enough CPU, memory, or time, or when environment setup differs between runs. Make environment assumptions and setup steps explicit. Google’s guidance notes that hermetic environments are generally less prone to flakiness (Google Testing Blog, 2021).
Choose a fix that restores determinism
Start from known state
Prefer tests that can run in any order and begin with known data. Rebuilding a fixture or database state for each test can be easier to reason about than cleanup, because cleanup may be skipped after a failure. The tradeoff is setup cost: rebuilding a large fixture can take longer. Choose based on the cost of setup and the risk of residue, and make the chosen lifecycle explicit.
Control time and synchronization
Put clock access behind a controllable seam where practical, then set or freeze time in tests that depend on dates or durations. For asynchronous behavior, wait for a meaningful state transition and fail with a bounded timeout and diagnostic details. Do not use an arbitrary delay as a substitute for knowing that the required state has been reached.
Use doubles deliberately
A test double can replace a remote or variable dependency when the goal is repeatable regression coverage. This improves control but reduces direct fidelity to the production interaction. Where that interaction matters, complement the double with contract or integration checks that verify important behavior against the real dependency. The right balance depends on whether a particular test is meant to isolate application logic or validate an end-to-end interaction.
Free tools Windows power users keep installed
One-click scans. No signup required.
Make the environment reproducible
Document and automate prerequisites, data setup, and runner assumptions. Compare the failing and passing runs for differences in resource allocation, environment variables, service readiness, or concurrent load. If a test only fails under resource pressure, address the underlying capacity or contention rather than masking it with a longer delay.
Use retries and quarantine as temporary controls
A retry can help identify an intermittent failure or keep a workflow moving, but a passing retry does not establish that the code is correct. Keep the original failure visible in logs and track the test as a reliability issue with an owner and a repair plan.
Rank #4
Quarantine can protect the main suite’s signal when a test is disrupting delivery, but it should remain visible, time-bounded, and scheduled for repair. Martin Fowler warns that quarantined tests can otherwise be abandoned (“Eradicating Non-Determinism in Tests”). Choose among immediate failure, retry, and quarantine by weighing pipeline continuity against diagnostic signal; no one policy fits every suite.
Historical figures are not current industry rates
Google reported that about 1.5% of its test runs were flaky and almost 16% of its tests had some level of flakiness in a 2016 article (Google Testing Blog, 2016). Those are historical figures from Google’s test corpus, not current measurements or industry-wide estimates. Google later described a continuous integration system running around 4.2 million tests in 2017 (Google Testing Blog, 2017); that figure is likewise specific to Google and its publication date.
Or skip the browser setup
If browser-based tests need screenshots to diagnose inconsistent page states, ScreenshotNeo offers a one-request screenshot API. It accepts the consent banner before capture and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each step can be turned off. Only clean shots are billed: bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits cost nothing, with the outcome identified in response headers. Its MCP server provides screenshot tools for AI agents. The free plan includes 1,000 shots a month with no card; paid plans start at $5 for 3,000 shots.
For request options and setup, see the ScreenshotNeo documentation. This cURL example saves a WebP screenshot:
Best Value
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpScreenshotNeo can remove common overlays before the shot, and its verdict and billing headers distinguish clean captures from failures and cache hits. Sign up for 1,000 free screenshots a month, with no card required.
Frequently asked questions
Does a test that passes on rerun prove the first failure was a false alarm?
No. A passing rerun shows that outcomes may vary; investigate the initial failure before deciding what it means.
Should every external dependency be mocked?
No. Use doubles to control dependencies when repeatable isolated coverage is the goal, and retain appropriate integration or contract checks when real interactions need validation.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
