October DealsAmazon USOctober deal check: compare before you payAmazon US: current deals, useful picks and tech finds.Check DealsSlow PC?RecommendedPC slow today? Run a repair scan before it gets worseResolve common Windows issues and optimize system performance.Scan NowOctober DealsAmazon USDeal season is back - check today's better picksAmazon US: current deals, useful picks and tech finds.See Picks×
Skip to content
World desk7 min

How to Manage Tests in a Continuous Integration Pipeline

A practical guide to CI test placement, merge-blocking checks, parallel jobs, suite health, and flaky-test recovery.
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Manage CI tests as a sequence of increasingly broad, increasingly costly checks: run fast, relevant tests early, then add integration, system, and end-to-end coverage where the extra confidence justifies the time and runner capacity. Keep flaky tests visible and owned; a retry that passes is not proof that a failure can be ignored.

Choose tests by the risk they detect

Start with the lowest test level that can reliably detect the behavior at issue. A unit test is usually the right place to check a small piece of logic. Integration tests cover interactions between components; system tests exercise larger assembled behavior; end-to-end tests cover complete user-facing journeys. Broader tests are valuable when the risk crosses boundaries, but they tend to cost more to run and maintain.

As an Amazon Associate I earn from qualifying purchases.

Most of a suite should generally be at the unit level, with fewer tests at higher levels. That is a directional principle, not a mandatory ratio: the right balance depends on architecture, failure risks, and how reliably each level catches defects in your codebase. GitLab’s testing-level guidance illustrates the distinction with its own inventory, estimated February 3, 2025: 218,459 unit tests (75.66%), 57,127 integration tests (19.79%), 12,444 system or feature tests (4.31%), and 704 end-to-end tests (0.24%) across GitLab Community and Enterprise Edition suites. Those counts describe GitLab, not an industry benchmark or target for other teams.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Place checks where their feedback is useful

Organize pipeline jobs so developers get quick, relevant feedback before waiting on wider coverage. A typical progression might look like this; treat it as a starting design, not a universal standard.

  1. Merge request: run formatting, static checks, and fast tests that directly cover changed code. Make dependable, high-signal checks blocking when a failure should prevent the change from merging.
  2. Broader validation: add integration and system tests for component interactions and assembled behavior. Choose whether these block merge or run in a later tier according to the risk they cover and the cost of delaying feedback.
  3. Deployment boundary: run focused smoke checks to establish that critical paths work in the deployed environment.
  4. Later or scheduled pipeline: run wider end-to-end coverage, longer suites, or checks whose value does not justify slowing every merge request.

GitLab’s documented testing strategy uses merge-request pipelines for unit tests, broadens integration and system coverage in later tiers, and reserves full end-to-end checks for selected higher tiers or scheduled pipelines; its deployment example includes smoke checks. This is one organization’s model. Map equivalent stages to your repository’s change risks and feedback needs rather than copying its exact tiers. GitLab Testing Strategy describes the approach.

Decide what should block a merge

A check belongs on the blocking path when it is both important to the change’s risk and dependable enough that engineers can act on its result. For each proposed job, consider the same decision questions:

  • How quickly will the result reach the author?
  • What kind of failure risk does the test detect, and is that risk relevant to this change or release?
  • How often does the job produce false alarms or fail because of unstable test or runner conditions?
  • What runtime and infrastructure capacity does it consume?
  • Who investigates failures, and what is the response when a check fails?
  • Should it block merge, deployment, or release—or report without blocking?

There is no universal acceptable CI duration, flake rate, retry count, or coverage percentage established by the cited guidance. Set thresholds from your release risk and baseline suite behavior, then review them when the codebase or infrastructure changes. Do not treat coverage percentage alone as evidence of test quality: it measures execution coverage, not whether assertions detect the failures your team cares about.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Speed up a slow pipeline without hiding work

Find the actual bottleneck

Measure job and suite elapsed times before changing the pipeline. Identify which suites dominate the wait and whether their work can be divided evenly. Optimizing a small job will not materially improve the end-to-end wait if another job remains the critical path.

Run independent jobs concurrently

CI systems can run independent jobs in parallel; dependencies still need to run in the required order. GitLab describes stages and jobs, with jobs in a stage able to run concurrently, while GitHub Actions jobs may run sequentially or in parallel. The exact configuration syntax depends on the CI platform. See GitLab CI/CD pipelines and GitHub Actions workflow concepts.

Split large suites into shards

If the runner and test framework support it, divide a large test job into smaller parallel jobs. GitLab supports parallel jobs with the parallel keyword and documents an RSpec example in Control how jobs run. Before adopting sharding, check that test distribution is effective, all shards contribute results to a complete report, and failures remain attributable to the tests that produced them. Compare the reduction in elapsed time with added runner use; parallelism shifts work across capacity rather than eliminating it.

Keep the feedback loop trustworthy

After each change, compare pipeline elapsed time, runner consumption, test-result completeness, and whether failures are still easy to diagnose. Avoid dropping a slow check solely to improve a dashboard number: first decide what risk it covers and whether that risk is covered elsewhere or can be tested at a cheaper level.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Handle flaky tests as defects to resolve

GitLab defines a flaky test as one that is unreliable, fails occasionally, and eventually passes if retried enough. Causes can include a brittle test, unstable infrastructure, or an unstable application. GitLab’s handbook warns: “Flaky tests undermine test results, leading to engineers disregarding test failures as flaky.” See Flaky tests.

A retry that passes is diagnostic evidence, not proof that the change is safe. For an intermittent failure:

  1. Preserve the original failure details, including logs and the conditions under which it occurred.
  2. Reproduce it where practical; compare the failing and passing environments and look for timing, ordering, data, or infrastructure differences.
  3. Determine whether the source is the test, the infrastructure, or the application, and assign a person or team to the follow-up.
  4. If it must be quarantined to keep unrelated work moving, track that state and its owner. Keep the test out of trusted blocking results only as a managed temporary measure.
  5. Fix it promptly and monitor it until it demonstrates stable behavior before returning it to the blocking suite.

GitLab’s pipeline-triage guidance describes quarantining flaky tests until proven stable, fixing them as soon as possible, and monitoring them until fixed. Quarantine without a tracked return path becomes silent loss of coverage. See Pipeline triage.

Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Make suite health an explicit responsibility

Test stages need ongoing maintenance, not just an initial YAML design. Review which tests provide distinct risk coverage, which are redundant, where runtime is growing, and whether owners respond to unstable results. Make changes to blocking rules, stage placement, and quarantine status explicit decisions so that faster feedback does not quietly weaken confidence in a release.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

GitLab’s strategy emphasizes fast feedback, progressive testing, resource efficiency, ownership, and stability. Its wording for the feedback principle is: “Fast Feedback Prioritize speed by running the most relevant tests first – fail fast, fix fast.” The useful principle is to put relevant checks early, while retaining wider tests where they protect against risks that narrow checks cannot catch.

Or skip the browser setup

If a CI job also needs website screenshots, ScreenshotNeo is a screenshot API and MCP server for developers. A single GET request captures a URL as PNG, JPEG, WebP, or PDF. For example, use this cURL request in a script or CI job:

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

See the ScreenshotNeo API documentation for request options. ScreenshotNeo accepts cookie and consent banners before capture and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each step can be turned off. Bot checks, blank pages, timeouts, failed loads, and cache hits cost nothing, and response headers report the page verdict and billing status. Its MCP server offers take_screenshot, get_page_info, and capture_pdf for AI agents. The Free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000 shots. Sign up for ScreenshotNeo’s free plan.

Common pipeline problems and fixes

  • Merge requests wait on broad suites: identify which jobs are on the critical path; move checks that do not need to block every change to a later tier, while preserving appropriate coverage for deployment or scheduled runs.
  • More parallel jobs do not shorten the wait: check whether the work is actually divisible and evenly distributed, and whether a dependent job remains the bottleneck. Compare elapsed-time gains against additional runner use.
  • A retry turns red into green: retain and investigate the first failure. A passing retry does not make the test reliable; determine the cause and track an owner.
  • Quarantined tests stay out of the suite: give quarantine an owner and tracked repair and requalification path, then monitor the test until it is stable.
  • A high coverage percentage still misses regressions: review whether tests assert the risky behavior and whether coverage is at an appropriate test level; do not use coverage alone as a quality verdict.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Leave a Reply

Your email address will not be published. Required fields are marked *

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

More from the Wire

  1. World desk4 min
    How to Spot an AI Voice Scam Before Sending MoneyDon’t rely on how a caller sounds. Pause, call back through a known number, and verify the emergency with another trusted person before sending money.
  2. Mountain View desk4 min
    Google’s SynthID Detector: How to Check AI-Generated Images, Video and AudioGoogle’s SynthID Detector looks for an embedded watermark in supported images, video and audio. Here is what its results do—and do not—show.
  3. Redmond desk20 min
    How to create a link to File or Folder in Windows 11Windows 11 gives you several ways to point to a file or folder without moving or duplicating it. You can create a desktop shortcut,…
Recommended PC Tool
Recommended PC Tool
Crashes, No Sound, or Screen Glitches?Free driver scan
Windows Errors? Fix Them Before They SpreadFree repair scan

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.