A consistently failing test gives a repeatable signal: something is wrong, though the cause may be in the test or the product. A flaky test can pass and fail on the same code under conditions meant to stay constant. That inconsistency can waste time with false alarms and, more seriously, teach a team to discount failures that may point to a real defect. Flakiness is therefore a risk to the reliability of the test signal—not proof that every intermittent failure is harmless or worse than every deterministic failure.
What makes a test flaky?
A flaky test produces different outcomes without an intentional change to the code or test conditions. For example, a test may pass in one CI run and fail in another on the same commit. That differs from a consistently failing test, whose failure is repeatable under comparable conditions.
As an Amazon Associate I earn from qualifying purchases.
The word “comparable” matters: a test can be sensitive to execution order, timing, environment, or external services. If those factors vary between runs, the results may vary too. An intermittent failure is not automatically a false alarm; a real defect can itself depend on timing or environment.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Why can flakiness be more dangerous than a repeatable failure?
A repeatable failure is disruptive, but it remains a clear signal that a test needs attention. A flaky test creates a two-sided problem: its false alarms consume investigation time, while its occasional passes can make the suite appear healthier than it is. If a team learns to rerun or ignore every failure from a noisy test, a failure caused by a genuine regression may be discounted as well.
- False alarms cost time. Developers may investigate a failure that does not recur, interrupting other work and delaying a pipeline.
- Repeated alarms weaken trust. When the same test often fails without an obvious product defect, people may stop treating its result as useful evidence.
- Intermittency can hide real faults. Microsoft Research cautions that failures from flaky tests may represent real faults in production code (Root Causing Flaky Tests in a Large-Scale Industrial Setting, 2019).
- CI disruption has wider effects. Mozilla’s developer-perspective research describes consequences for scheduling, resource allocation, and confidence in the test suite (developer-perspective study).
Meta’s engineering article captures one team’s framing of the asymmetry: “A passing test indicates the absence of corresponding regression, while a failure is merely a hint to run the test again.” That is an observation about the practice described in its 2020 article, not a rule that applies to every team or every test (Probabilistic flakiness: How do you test your tests?).
“More dangerous” is an operational comparison, not a universal ranking. A deterministic failure in a critical test can be more serious than a low-impact flaky test. Risk depends on the behavior under test, the cost of false alarms, and how the team handles intermittent results.
What causes flaky tests?
There is no single leading cause across all languages and organizations. Studies point to several recurring sources of variation:
- Test order and shared state: one test changes data or state that affects another, so results depend on execution order.
- Asynchronous behavior and timing: a test checks before an operation has completed, or depends on a timing assumption.
- Concurrency: competing operations can produce different outcomes or expose timing-sensitive defects.
- Infrastructure and environment: resource limits, configuration, or other environmental differences change execution.
- External dependencies and networks: a service or network response may vary or become unavailable.
- Randomness APIs: tests that use random values can encounter different inputs or states on different runs.
The proportions in individual studies are specific to their samples, not industry-wide estimates:
| Study | What it found | Scope |
|---|---|---|
| An Empirical Study of Flaky Tests in Python (2021) | Of 7,571 flaky tests, 59% were attributed to order dependency and 28% to test infrastructure; most of the remainder involved network and randomness APIs. The authors estimated an average of 170 reruns to reach 95% confidence that a passing test case was not flaky. | Analysis of 22,352 projects and 876,186 test cases. The rerun estimate is specific to the paper’s method and dataset, not a universal recommendation. |
| A Study on the Lifecycle of Flaky Tests (2020) | Asynchronous calls were the leading cause in the studied projects. Some developer-reported fixes did not empirically reduce flakiness. | Six large-scale proprietary Microsoft projects. |
| Root Causing Flaky Tests in a Large-Scale Industrial Setting (2019) | Identified concurrency and external dependencies among causes. | Microsoft industrial setting; the publication does not establish a universal distribution of causes. |
How do you tell a flaky test from a real failure?
One pass after a failure is evidence of variation, not proof that the application is correct or that the first failure was benign. Keep the failing result and compare it with passing runs before drawing a conclusion.
- Preserve both outcomes. Record the commit or code version, test name, full failure output, timestamp, and whether the result came from an initial run or a retry.
- Compare execution context. Check test order, environment and configuration, timing, concurrency, infrastructure state, and external-service or network behavior.
- Reproduce the failure in context. Try the same environment and relevant ordering or timing conditions. A failure that disappears in a different setup has not necessarily been explained.
- Investigate the product as well as the test. A timing-sensitive or environment-dependent regression can be intermittent. Do not classify a failure as harmless solely because it passed once on rerun.
- Verify the proposed fix with repeated observations. Compare outcomes before and after the change under relevant conditions. A fix label or a green build alone does not establish that failure frequency fell.
Google’s De-Flake Your Tests work describes comparing runtime information from passing and failing executions. Across its reported case studies, the approach located root causes with 82% accuracy; that figure applies to those case studies, not to every test suite or tool.
Rank #4
Do reruns solve the problem?
Reruns can reveal some intermittent outcomes, but they do not guarantee that a test is reliable or that a failure can be ignored. A 2021 Python study estimated an average of 170 reruns for 95% confidence that a passing test case was not flaky; that dataset- and method-specific result illustrates why a small number of passes is not conclusive, rather than setting a standard rerun count.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsEvidence from CI pipelines also shows the limits of routine retries. A 2026 accepted/in-press study by Leinen, Gruber, Erdogan, Stahlbauer, and Pretschner analyzed 8.8 billion test executions from four industry-scale projects over two-month periods. It reported that 9.8%–16.3% of failed pipeline runs involved undetected flaky failures, and that flake rates varied by as much as 3× between environments. These results apply to those projects and periods, not to CI pipelines generally (study record).
Best Value
A retry can be useful operationally if it reduces immediate disruption, but keep the original failure visible in the run history. If a pipeline turns green after retrying, distinguish that from a clean first-run pass. Quarantine may also limit disruption, provided the failure remains tracked with an owner and is not silently excluded from release decisions.
How should teams fix flaky tests?
Start with the observed cause rather than a generic retry policy. If results depend on order, isolate test state; if timing is involved, make the test synchronize with the condition it actually needs; if an external dependency is involved, examine the dependency and how the test handles its availability. The right change depends on the failure evidence.
- Keep failing and passing run details so the comparison remains possible.
- Track flaky tests and assign responsibility for investigating them, even when retries or quarantine are used to manage immediate CI disruption.
- Evaluate a fix by observing whether the failure frequency changes under relevant conditions.
- Do not treat a passing retry as evidence that the tested behavior is correct.
The Microsoft lifecycle study found examples where developers believed they had fixed a flaky test but experiments did not show a reduction in flakiness. Verification matters because a change can make a test pass once without removing the source of variation.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
