DriversRecommendedOutdated drivers can make a good PC feel brokenScan driver issues before chasing fixes manually.Scan NowOctober DealsAmazon USOctober deal check: compare before you payAmazon US: current deals, useful picks and tech finds.Check DealsPC HealthRecommendedCrashes, freezes, slowdowns? Check your PC nowSpot repairable issues before they interrupt work.Check PC×
Skip to content
World desk8 min

How to Cut Regression Testing from Weeks to Days

A staged approach to shortening regression testing: measure where time goes, remove execution waste, prioritize likely failures, then select and budget tests with an explicit risk choice.
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

For most teams, the realistic path from weeks to days is a staged one. First remove waste from how the suite runs, then put likely failures earlier in the run, and only after that decide which tests to skip for a given change. Skipping tests is the last lever to pull, because it trades speed for coverage you must then account for. Published examples of “weeks to hours” exist, but they describe specific systems and should not be read as a normal outcome.

First, confirm what is taking weeks

“Regression testing takes weeks” can mean three different things, and each calls for a different fix. It may be one automated run that takes days, a queue where runs wait behind each other for days, or a manual regression cycle run by people. The tactics below target automated suites in continuous integration (CI). In a manual cycle, the main lever is usually deciding what must be checked at all, and automation only helps where automated checks are already part of that cycle.

As an Amazon Associate I earn from qualifying purchases.

Before changing anything, write down which of the three you have. A suite that runs in four hours but waits three days for a free worker has a scheduling problem, not a test-selection problem.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Selection and prioritization solve different problems

Most of the confusion in this topic comes from using “selection” and “prioritization” as if they were the same. Selection decides which tests run at all. Prioritization decides the order in which the chosen tests run. Selection changes membership; prioritization changes sequence. Google’s 2014 work on regression testing in continuous integration describes selecting tests before a change is submitted and testing dependent modules after submission, which is one way of splitting the work across the pipeline (Google Research publication record).

Lever What it decides How it speeds feedback What it costs
Selection Which tests run for this change Runs fewer tests, so the run ends sooner Tests not selected are delayed, and a missed dependency can hide a real failure
Prioritization Which selected tests run first Brings likely failures forward, so the first failure appears sooner Does not by itself shorten total run time if the whole suite still runs

Combining the two can make feedback faster, but it means some checks happen later or outside the immediate change workflow. That is a deliberate trade, and it needs an owner.

A staged sequence for cutting the time

1. Establish a baseline

Record the measures that tell you where time goes:

  • Wall-clock duration of the full run, and of the slowest stage
  • Queue time before execution starts
  • Execution time per test, separating tests that are slow themselves from tests waiting on shared databases, environments, or other serialized resources
  • Time to first useful failure
  • Total test count, failure rate, flakiness, and the areas of the product each test covers

Microsoft’s Azure well-architected testing guidance recommends monitoring execution-time trends and test reliability measures over time (Microsoft Learn, Azure testing guidance). Shopify’s engineering team uses time to first failure, alongside fault-detection measures, to judge its prioritization (Shopify Engineering, March 7, 2022).

2. Remove avoidable execution waste

AWS’s DevOps guidance recommends these steps before adopting machine learning (ML) based test selection, and this order is the one most worth following:

What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

“Before choosing to implement advanced test selection methods using machine learning (ML). you should first optimize test execution through parallelization, reducing stale or ineffective tests, improving the infrastructure the tests are run on, and changing the order of tests to optimize for faster feedback.”

In practice that means:

  • Parallel execution for tests that are independent and an environment that can absorb concurrent workers.
  • Better infrastructure if worker scarcity, slow setup, or cold environments dominate the timeline.
  • Suite hygiene: remove obsolete and duplicate tests, and review those that no longer detect anything meaningful. Do not delete a test just because it is slow; check what behavior and risk it covers first.
  • Repair flaky tests rather than rerunning them until they pass.

Parallelism shortens elapsed time without reducing the number of tests, but it does not remove shared state or hidden dependencies. A test that passes alone may fail when it runs beside another test that changes the same data. A 2020 paper on dependent-test-aware regression testing warns that dependence can contribute to flaky failures when tests are reordered, selected, or parallelized (University of Washington, ISSTA 2020 abstract). Partition work only after you have identified which tests share state.

3. Select tests related to the change

Change-based test impact analysis looks at the code difference and identifies tests likely to exercise the modified code. AWS describes this as a structured way to run a relevant subset without ML (AWS DevOps Guidance). The map from code to tests has to be maintained as the architecture and coverage change. A test left out of a run is delayed, not proven irrelevant forever, so the selection rules need a scheduled review and a broader run that catches what they miss.

4. Prioritize the selected tests

Within the selected set, run tests with a stronger history of failure, or closer relevance to the change, first. A failing change then surfaces its problem earlier, even if the full set still takes the same time to finish. Shopify’s approach layered a history-based prioritized set on top of change-based selection and measured performance under fixed time limits (Shopify Engineering, 2022).

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

5. Add a time budget only after measuring

A time budget stops a prioritized run at a chosen limit. The limit should come from observed distributions and an explicit view of acceptable risk, not from an arbitrary target such as “under 20 minutes.” Shopify’s 2022 analysis reported the following for its own system:

  • In its mean-case analysis, ordering by failure rate found 80% of failures after running 60% of the selected tests.
  • In its 5th-percentile view, 70% of the selected suite found 50% of failures.
  • The selected suite was a median 40% of the full test suite.

These figures describe Shopify’s large monolith and its data. They are not a service-level promise for another codebase. To estimate your own numbers, replay recent history or run the budget in shadow mode alongside the full suite, then compare time to first failure, the share of failures detected, and the percentage of tests run.

6. Keep a slower full-suite safety net

Use the fast checks for frequent feedback, and keep the broad, slow runs on a separate cadence. Microsoft’s Azure guidance recommends nightly full-suite runs in pre-production for long-running tests, along with fail-fast handling for critical tests. AWS recommends running a full set asynchronously when predictive test selection is used. It also cautions against excluding security tests from selection, and against relying on predictive selection for sensitive, critical systems.

7. Read results as quality signals

Track execution-time trend next to pass rate, flakiness, and defect escape rate, meaning bugs that reach production and that the suite should have caught. When an escape happens, add or correct a regression test at the point where the gap occurred. Avoid making coverage percentage the only target. Azure’s guidance treats it as one signal and asks teams to emphasize high-risk paths.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Choosing among the approaches

The approaches are not alternatives to one another. Most teams combine several, in the order above. The table compares them on the axes that usually decide whether a change is worth making.

Approach What it changes Best when Main caution
Parallel execution Runs independent tests concurrently Total wall time is high and workers and environments can scale Shared state and dependencies can make parallel runs unreliable; watch resource contention
Suite cleanup Removes stale or duplicate tests, repairs flaky ones The suite has accumulated test debt or low-signal checks Verify what risk a test covers before removing it, even if it is slow
Test ordering Runs likely failures earlier The full suite must still run but feedback should arrive sooner Ordering alone does not necessarily reduce total completion time; measure time to first failure
Change-based selection (test impact analysis) Chooses tests related to modified code Code-to-test relationships are available and can be maintained Missed dependencies can omit relevant checks; keep broader runs
Predictive selection Uses historical changes and results to predict relevant tests Historical data is available and the risk can be governed Model uncertainty; AWS advises controls and warns against using it for sensitive critical systems
Time-budgeted selection with prioritization Stops a prioritized run at a chosen limit Failure yield can be quantified and an explicit risk accepted A locally chosen budget may miss failures; retain full-suite coverage elsewhere and reassess

What published “weeks to hours” results show

CaseStudies.com hosts a case study attributed to Perfecto that describes a large North American bank cutting regression testing for a 2,000-test suite from two weeks to seven hours. The reported approach combined code optimization with parallel execution, and automated coverage was stated as roughly 70% per release. Three limits matter. The bank is unnamed, the account is vendor-attributed, and the page does not give a date. It is best read as evidence that the staged approach can produce large gains in one environment, not as a benchmark you should expect to match (CaseStudies.com, Perfecto case study).

Academic results are more modest and more specific. A 2015 industrial case study reported 79.5% execution-cost savings with fault-detection capability above 70% for test-suite minimization using finer-grained coverage. The same study found test-selection savings below 2%, which shows how much results depend on the system and the changes involved (Wiley, Software Testing, Verification and Reliability, 2015).

Flaky tests undermine every speed gain

Flaky tests pass or fail nondeterministically on the same code, and they make every faster pipeline less trustworthy. Microsoft Research’s 2020 study on the lifecycle of flaky tests describes the problem plainly: “flaky tests, which nondeterministically pass or fail on the same code, are problematic because they provide misleading signals during regression testing.” The study found asynchronous calls were a leading cause across six studied Microsoft projects (Microsoft Research, 2020).

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

The same paper’s proposed FaTB approach reduced runtimes by up to 78% on five tests affected by asynchronous calls. The evaluation covered only those five tests, and the paper reports no empirical change to how often their flaky failures occurred. Treat it as a sign that targeted diagnosis of slow, async-dependent tests can pay off, not as an expected speedup for your suite.

When a test flakes, diagnose it before changing the schedule around it. Rerunning until it passes adds time and hides the signal you need.

Where the 78% and the 80% figures stop applying

Each published figure in this topic comes from a single system, a single dataset, or a single vendor. Before quoting one to a stakeholder, check three things: whether it measured time or failure yield, whether it was a mean or a conservative percentile, and whether the team already had the historical data it depends on. A figure that holds for a monolith with years of failure history may not hold for a young service with few runs. That is the reason the sequence above starts with measurement.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Leave a Reply

Your email address will not be published. Required fields are marked *

Free tools Windows power users keep installed

One-click scans. No signup required.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

More from the Wire

  1. World desk4 min
    How to Spot an AI Voice Scam Before Sending MoneyDon’t rely on how a caller sounds. Pause, call back through a known number, and verify the emergency with another trusted person before sending money.
  2. Mountain View desk4 min
    Google’s SynthID Detector: How to Check AI-Generated Images, Video and AudioGoogle’s SynthID Detector looks for an embedded watermark in supported images, video and audio. Here is what its results do—and do not—show.
  3. Redmond desk20 min
    How to create a link to File or Folder in Windows 11Windows 11 gives you several ways to point to a file or folder without moving or duplicating it. You can create a desktop shortcut,…
Recommended PC Tool
Recommended PC Tool
Crashes, No Sound, or Screen Glitches?Free driver scan
PC Slower Than It Used to Be?Free scan - under a minute

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.