Flaky tests persist partly because investigating one can be a costly bet: the failure may come from the test, the environment, or a real product bug, and reproducing it can take time. When clearer, deadline-driven work competes for the same engineering hours, postponing the investigation can make sense locally—even as repeated reruns and unreliable signals impose costs on the whole team. That is a plausible explanation supported by documented testing challenges, not proof that every team ignores flaky tests or that incentives alone cause them to persist.
What makes a test flaky?
A flaky test can pass and fail on the same code. Its result is therefore a noisy signal: a failure may indicate a regression, but it may also be an intermittent test or environment problem. Treating every failure as definitive can trigger unnecessary investigations; treating intermittent failures as harmless can hide a real fault.
As an Amazon Associate I earn from qualifying purchases.
There is no single universal cause. Asynchronous work, timing, concurrency, nondeterministic or undefined behavior, third-party dependencies, and variation in infrastructure or hardware can all contribute. A 2023 literature review also describes nondeterminism in machine-learning applications. These categories can overlap; identifying a plausible category is not the same as diagnosing a particular test.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
What studies found about causes
In a study of six proprietary projects, Microsoft Research found asynchronous calls were the leading cause among the flaky tests it examined. That finding describes those projects, not every software team. Microsoft’s industrial root-cause study also discusses concurrency and external dependencies, while Google’s practitioner guidance names concurrency, nondeterministic or undefined behavior, third-party code, and infrastructure problems.
Why deferring an investigation can seem rational
When a test fails intermittently, the cause is uncertain. An engineer may need repeated runs to reproduce the failure, then more analysis to decide whether the test, recent code, or the execution environment is responsible. If the failure is unrelated to the latest change, the investigation still consumes time. Google’s 2023 work on flaky-test culprit finding describes costly reruns as part of failure attribution.
That creates a short-term trade-off. A team under schedule pressure may defer a difficult diagnosis in favor of product work with a clearer immediate payoff. This is a reasoned account of how documented uncertainty, effort, and scheduling pressures can shape decisions; the studies cited here do not directly measure how often teams make that choice or prove that incentives cause flakiness to persist.
The local choice can nevertheless add up to a collective cost. Reruns consume CI capacity and developer attention; inconsistent outcomes make it harder to tell whether a change broke the product. In a 2019 study involving 21 professional developers and 200 previously fixed flaky tests, Mozilla researchers documented developers’ perceptions of effects on resource allocation, scheduling, and confidence in test suites. Microsoft’s 2022 account said its flaky-test management system was being used by more than 100 product teams. These figures describe the studies and system reported, not industry-wide rates.
Why a passing rerun is not a fix
A rerun that passes tells you the failure did not recur on that run. It does not establish that the underlying cause has gone away. The test may pass by chance, or the triggering conditions may not have occurred.
Microsoft Research’s lifecycle study found cases in which developers said they had fixed a flaky test, but empirical experiments did not find that their changes reduced the tests’ failure frequency. This is why a claimed repair should be checked against repeated runs or other diagnostic evidence, rather than accepted on the strength of one green result.
The same study evaluated FaTB on five flaky tests and reported runtime reductions of up to 78% without empirically affecting the frequency of flaky failures. That result concerns runtime, not a demonstrated fix for flakiness; it should not be read as evidence that faster execution resolves intermittent failures.
Rank #4
Can a flaky failure be a real bug?
Yes. An intermittent failure can still expose a fault in production code. Microsoft’s industrial study warns: “However, ignoring failures of flaky tests can be dangerous, since those failures may represent real faults in the production code.” A test’s history of inconsistency is a reason to investigate its signal carefully, not proof that a particular failure is false.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsRerunning may help gather evidence, but it should not erase the original failure from consideration. If the failure recurs under particular conditions, or points to a recent change, that evidence matters even if a later run passes.
Best Value
How to respond without mistaking silence for reliability
The evidence supports judging responses by whether they improve diagnosis and ownership—not merely whether the build turns green.
- Use reruns to gather evidence. Record the original failure and the rerun outcomes. A pass may help characterize intermittency, but it does not by itself identify a cause or verify a fix.
- Look for diagnostic evidence. Consider timing, asynchronous work, concurrency, dependencies, and environment variation as possible leads rather than assuming one root cause. Confirm a repair with repeated runs or evidence tied to the suspected cause.
- Make any quarantine or deferral owned and visible. Assign responsibility and a path back to diagnosis. Suppressing a failure without follow-up risks hiding evidence of a real fault and lets the unreliable signal persist.
- Account for shared costs. Repeated reruns and investigations draw on CI and engineering time. Making that friction visible helps teams weigh a repair against other scheduled work instead of treating the failure as free noise.
The practical goal is not to treat every red run as a confirmed regression, nor to dismiss intermittent failures. It is to preserve the evidence, establish who will investigate, and verify that a claimed fix actually changes the failure pattern.
What the evidence does—and does not—establish
The studies describe costly reproduction and attribution, varied technical causes, and effects on scheduling, resource allocation, and confidence in test results. They give a grounded basis for explaining why a team might defer a difficult investigation and why that deferral can have wider costs. They do not establish that everyone ignores flaky tests, quantify a universal cost of flakiness, or measure a general causal link between organizational incentives and persistence.
Recommended Free Tools
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
