Don’t decide a broken test suite’s fate by its failure count alone. First identify why it fails, then ask whether each test provides distinct confidence that the product behaves correctly. Refactor useful checks whose implementation is unreliable; rebuild when the suite’s structure makes repair more costly than replacement; delete tests that detect no meaningful defects and add only maintenance. There is no universal failure-rate or time threshold for choosing among the three.
Diagnose the failure before changing the suite
A test that passes and fails without a noticeable change to code, tests, or environment is nondeterministic. Re-running it can confirm the symptom, but repeated runs do not fix its cause. A failure may originate in the test, the runner or framework, the application and its dependencies, or the operating system, hardware, and network. Google’s 2021 flakiness guide details causes across those layers.
As an Amazon Associate I earn from qualifying purchases.
Gather evidence before rewriting assertions or loosening timeouts. Look for setup and cleanup gaps, shared or stale data, order dependencies, assumptions about time, races in asynchronous behavior, resource starvation, runner collisions, dependency changes, and infrastructure errors. Compare failed and successful runs, inspect relevant runner and system logs, and check what changed in the application, test code, dependencies, and environment.
- Failure follows a test or its order: run it alone and in different orders; inspect state initialization, cleanup, and shared data.
- Failure varies with timing: examine race conditions, synchronization, timeouts, and resource contention.
- Several unrelated tests fail together: investigate runner capacity, network and service availability, disk errors, and other processes competing for resources.
- Failure follows a dependency or application change: check whether the behavior changed, the test depends on an unstable interface, or a service’s contract was altered.
Match the remedy to the cause. Establish a known starting state, isolate tests, fix infrastructure faults, or synchronize on the expected application state. Avoid adding arbitrary sleeps: Google warns that they can become flaky again and unnecessarily slow execution.
Decide whether each test earns its place
Evaluate tests by the behavior they protect, not by how long they have existed or how many assertions they contain. For each check, identify the defect it should catch and whether another test already catches it with comparable confidence. A green suite should provide confidence that significant bugs are not present—not merely demonstrate that the tests ran. Martin Fowler describes that as the test of a test suite in his Continuous Integration article.
- Keep or improve it if it checks an important user-visible behavior, integration, or system property that would otherwise go untested.
- Move or narrow it if the same behavior can be checked more quickly and reliably at a lower test layer, without losing the integration assurance it uniquely provides.
- Delete it if it only mirrors implementation, breaks on harmless internal changes, duplicates other coverage, and contributes no distinct confidence.
Tests that merely detect implementation changes can have negative value. Alex Eagle writes that change-detector tests “do not catch any defects” while their maintenance slows development in Google’s 2015 discussion. Before deleting a test, ask what failure would go unnoticed afterward. If the answer is a real behavior or integration risk, replace or relocate the check rather than silently dropping the confidence it provided.
Refactor when the signal is valuable and the cause is fixable
Refactoring is the right first choice when a test checks behavior worth preserving and evidence points to a repairable problem: fragile setup, poor isolation, vague synchronization, noisy assertions, inadequate diagnostics, or an unsuitable test boundary. Fix the source of unreliability rather than hiding it with retries or a longer timeout.
- Make state predictable. Give the test a known starting state, use isolated or ephemeral data where practical, and clean up reliably.
- Remove order dependence. Run affected tests independently and in different orders to expose shared state or assumptions left by earlier tests.
- Synchronize on behavior. Wait for the expected application state or event rather than inserting arbitrary delays.
- Improve failure evidence. Preserve useful logs and relevant state so a failure can be understood rather than merely reproduced.
- Verify the assertions. Confirm that the refactored test would still fail for the defect it is meant to catch. Google’s practical question is: “How do you know that your refactoring of the tests was safe and you didn’t accidentally remove one of the assertions?”
That final check matters: a test can become stable by accidentally becoming toothless. Retain meaningful behavior checks even when changing setup, synchronization, or the layer at which the behavior is tested.
Rebuild when accumulated design debt makes repair uneconomic
Consider rebuilding when feature work and maintenance have become cumbersome because of the suite’s underlying design—not simply because it has recently produced many failures. A replacement may make sense when repairing isolation, ownership, architecture, or execution patterns would cost more than establishing a better foundation. Conversely, a rewrite can discard valuable coverage and introduce new defects, so compare its likely cost and risk with targeted repairs.
There is no established universal failure percentage, elapsed time, or test-count threshold that says “rebuild.” Use your own evidence: time spent maintaining tests, time to diagnose failures, execution and resource costs, trust in the signal, ownership, and important coverage gaps. Martin Fowler’s discussion of testing culture frames rewrite decisions around whether the cost of a rewrite is becoming preferable to paying down accumulated debt; it does not prescribe a numeric cutoff. See “Goto Fail, Heartbleed, and Unit Testing Culture.”
Rank #4
If you choose a replacement, preserve the behavioral requirements and unique confidence of the old suite. Rebuild around those needs rather than translating every existing test unchanged: some checks may be redundant, some may belong at another layer, and some integration risks may require a deliberate end-to-end test.
Keep a small, intentional end-to-end layer
Many broad end-to-end tests can be slow, flaky, and expensive to maintain. That is a reason to be selective, not to abandon end-to-end coverage. Keep tests for important user journeys and system properties that smaller checks cannot reliably evaluate, such as resource allocation, concurrency, or API compatibility. Assert on overall behavior rather than volatile implementation details.
Best Value
Google’s 2016 end-to-end guidance recommends targeting important use cases, making tests diagnosable with overview logs and preserved state such as screenshots or database snapshots, and using ephemeral test data where possible. External services and other teams’ systems can reduce repeatability; fakes and stubs can also drift from real implementations. Account for those trade-offs when deciding which integration risks need a real end-to-end check.
Google’s article offers “at least one week per quarter per end-to-end test” as planning guidance for stabilizing tests affected by slow or flaky dependencies or minor UI changes. It is not a measured universal average or a rule that every team must spend that amount. The broader lesson is to budget for the upkeep of each test you decide is worth keeping.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Compare viable suite designs with evidence
If more than one design could preserve the needed confidence, compare them across speed, maintainability, resource utilization, reliability, and fidelity—the dimensions Google groups as SMURF in its 2024 roundup. Add the measures that reflect your team’s risks:
Recommended Free Tools
- Unique confidence: Which defects or integration failures does each test layer detect that another does not?
- Diagnosis time: How quickly can an engineer identify the cause of a failure?
- Ownership: Is there a clear team or person responsible for fixing tests and maintaining their dependencies?
- Coverage of integration risks: Are important interactions exercised somewhere, even if most checks move to smaller, faster tests?
Use the test pyramid as a heuristic, not a quota. Fowler’s Practical Test Pyramid favors many small, fast checks, some broader tests, and few high-level tests, while emphasizing the need to keep high-level checks that provide unique confidence. The right balance depends on the system’s risks and on whether a test faithfully exercises the behavior it claims to cover.
A practical decision sequence
- Triage. Identify whether failures are deterministic, isolate the relevant failure modes, and inspect test, runner, application, dependency, and infrastructure evidence.
- Map confidence. For each test, name the behavior or risk it covers and find duplicates or gaps across the suite.
- Refactor valuable checks. Repair root causes and verify that meaningful assertions still detect the intended defects.
- Delete checks with no distinct value. Remove change detectors and redundant coverage only after confirming they do not protect a unique behavior or integration risk.
- Compare repair with replacement. Rebuild only when measured maintenance burden and structural debt make targeted repair the less sensible investment.
- Reassess the balance. Keep enough broad testing to cover important system behavior, but make every expensive test justify its maintenance and execution cost.
Google’s older 2007 automation article describes the familiar costs of UI-heavy scripted testing—maintenance, slow execution, flakiness, and failures for the wrong reasons—and argues for smaller API-level checks where suitable without abandoning UI or end-to-end coverage. Treat that as design guidance, not a current benchmark: the decision is still about the right balance for your system.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
