Recommended Free Tools
A flaky test passes and fails under apparently unchanged code. The useful response is to preserve the failure, vary execution conditions to expose a pattern, and repair the cause—not to treat a successful retry as proof that the failure was harmless. Use these six steps to work from diagnosis to a controlled short-term mitigation.
1. Capture the failure before rerunning it
Start with the failing commit and preserve enough context to compare the failure with a passing run. Record the test name, complete logs, runner and environment details, test order, and nearby test results. A rerun can overwrite useful evidence or change the conditions that triggered the problem.
As an Amazon Associate I earn from qualifying purchases.
- Save the exact commit or build identifier and the failure output.
- Note the runner, operating system, relevant configuration, and whether the test ran alone or as part of a suite.
- Compare a failing execution with a passing one: identify the first point where their logs or behavior differ.
Rerun the test independently, then run it again in its original sequence. If it passes alone but fails in the suite, the preceding tests or shared process state become important leads. Chromium’s debugging guidance recommends examining tests that ran earlier in the same process and using logs or a debugger to find where execution diverges: Chromium: Fixing Flaky Unit Tests.
2. Vary one execution condition at a time
Once you can repeat the test, change one condition per run and record the result. A changed outcome is evidence about the failure pattern, not yet proof of its cause.
- Order: run the test first, last, or after a suspected predecessor.
- Process: compare an isolated process with the original suite or worker arrangement.
- Host: when possible, compare another CI runner or a local run with the CI environment.
- Time: where the system permits controlled time changes, check whether the result depends on time-related assumptions.
For example, a failure that follows a particular predecessor suggests order dependence or leaked state; one that appears only on a loaded host points toward scheduling or resource conditions. GitHub describes using retries under different conditions to help classify likely causes, but its reported results describe GitHub’s internal approach rather than a guarantee for other teams: GitHub: Reducing flaky builds by 18x.
3. Reset state and isolate tests
A test can pass alone and still fail in a suite if another test changes state it depends on. Martin Fowler’s guidance is to keep tests isolated so execution of one does not affect others: Eradicating Non-Determinism in Tests.
Inspect state that can outlive an individual test or leak across workers:
- Database rows, files, caches, and other persistent resources.
- Environment variables and process-wide configuration.
- Static data, singletons, shared fixtures, and global variables.
Make each test establish the state it requires, scope mutable state as narrowly as possible, and arrange cleanup so it runs even when an assertion fails. Chromium documents scoped state setters and checks for global state as useful debugging practices: Chromium’s flaky unit test guidance. pytest also identifies uncontrolled state and order dependence among causes of flaky tests: pytest: Flaky tests.
4. Replace timing guesses with meaningful synchronization
A fixed sleep assumes that an operation will finish within a particular duration. Under a different load or scheduler, it can be too short and fail—or unnecessarily long and slow every run. Instead, wait for a signal that represents the state the test actually needs, such as a completed response or a specific condition, with a bounded timeout. Include enough logging to tell whether the expected condition appeared and when.
Google’s 2021 triage guidance warns: “Do NOT add arbitrary delays as these can become flaky again over time and slow down the test unnecessarily.” Google Testing Blog: Test Flakiness, Part II.
Rank #4
5. Check the framework, dependencies, and CI runner
The test code is only one possible source. Google’s triage guidance separates failures originating in the test, the framework, the application or its dependencies, and the operating system or hardware. Follow the evidence across those boundaries rather than assuming every intermittent failure is a bad assertion.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →- Test: look for hidden state, order dependence, nondeterministic data, or timing assumptions.
- Framework: inspect scheduling, parallel execution, fixture lifetimes, and worker behavior.
- Application or dependency: check service availability, response timing, and whether the test has exposed a real race or regression.
- Runner or system: inspect network and disk errors, resource exhaustion, and differences between local and CI configurations.
A failure seen only under CI load may indicate constrained resources, but it may also reveal a genuine concurrency defect. Compare the failing and passing environments and correlate test logs with runner and dependency errors before deciding which it is. Google’s triage guidance provides this broader failure-source framework.
Best Value
6. Use retries and quarantine as managed controls
A retry that passes confirms inconsistent observed outcomes; it does not explain why they differ or establish that the initial failure was harmless. pytest warns that permanently quarantining flaky tests is dangerous because it can leave important failures unnoticed: pytest’s flaky-test documentation.
Use retries to gather evidence or limit disruption, and make their role explicit. A retry in identical conditions may show intermittence; a retry under a deliberately changed condition can help distinguish likely causes. Neither replaces a fix. If a test must be quarantined to protect the signal from the main suite, keep the change visible, assign an owner, and set a review or expiry limit so it returns to normal coverage after repair.
Prioritize recurring flakes by both frequency and impact: a rare failure in a critical deployment check can matter more than a frequent failure in a low-risk test. Google’s 2016 figures illustrate why flakiness can be consequential within a large test corpus, but they are historical measurements of Google’s own tests, not current industry rates: Google reported about 1.5% of test runs were flaky, almost 16% of tests had some level of flakiness, and about 84% of observed pass-to-fail transitions involved a flaky test. Google Testing Blog: Flaky Tests at Google and How We Mitigate Them. GitHub reported identifying 90% of flaky failures with its own retry approach; that is an account of an internal system, not an independent benchmark or a forecast of results elsewhere: GitHub’s account of its approach.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Keep a short evidence trail
For each recurring failure, track the test, failing and passing commits or runs, conditions changed, observed pattern, suspected cause, and owner. This prevents a retry from being mistaken for resolution and lets the team distinguish a test issue from a runner or application problem. The Flake project describes a CLI and GitHub Action for controlled variation and replay; those are project-described capabilities, not independently validated performance claims: Flake project repository.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
