Quarantine a flaky test only as a temporary, visible exception: save the failure evidence, link the test to an issue and accountable owner, remove it from the CI blocking path using your framework’s supported mechanism, and set a review date and exit plan. Quarantine is not a fix. Keep investigating until you repair, replace, remove, or deliberately reintroduce the test.
What quarantine does—and what it does not do
A flaky test does not produce a consistent result under the same intended conditions. A failure followed by a passing retry is evidence of inconsistency, not proof that the original failure was harmless. Possible causes include a brittle test, unstable test infrastructure, or an unstable application. GitLab describes quarantine as skipping a test in CI while preserving it in the codebase for future fixing; its handbook says quarantined tests run locally by default. See GitLab’s quarantine guide and flaky-test guidance.
As an Amazon Associate I earn from qualifying purchases.
Excluding a test from the blocking path reduces the chance that an intermittent failure blocks unrelated work, but it also means that test no longer provides the same CI gate. Keep its failure visible and its coverage need understood. Do not use quarantine to silently suppress a known product regression: classify the failure and make the reason explicit.
Confirm the failure and preserve evidence
Before muting the CI signal, establish what failed and retain enough context for another engineer to reproduce or investigate it. A retry may help characterize the pattern, but it does not by itself identify whether the cause lies in the test, environment, or application.
#1 Best Overall
- Record the failing pipeline or job and the test’s exact identity.
- Keep the stack trace, failure pattern, and relevant logs or artifacts.
- Capture the random seed, environment, and other run details needed to reproduce the failure, when available.
- Open or link an issue describing the observed behavior and why the test needs quarantine; apply the appropriate failure label if your process uses one.
GitLab’s debugging guidance suggests reproducing with the CI seed, bisecting when useful, and checking for state leakage and order dependence. Those are investigation ideas, not universal commands: use the test runner’s documented options for your own repository. After a fix, rerun the test in the conditions that exposed the failure.
Decide whether quarantine is justified
Fix the cause directly when it is understood and can be addressed promptly. Quarantine is more appropriate when a failure is blocking important development and a reliable fix cannot be delivered promptly, provided the exception remains tracked and owned. If the cause is unknown, make investigation itself an explicit task rather than treating the quarantine as resolution.
- Urgency: Is the test blocking critical work now, or can the team investigate before changing the gate?
- Cause: Is the problem understood and readily repairable, or does it need deeper investigation?
- Follow-up capacity: Can the team meet a short review deadline, or does this need a longer tracked process?
- Coverage impact: Can you repair the test, replace it with better coverage, or remove it without losing an important regression check?
State the reason in the issue and quarantine metadata. GitLab’s implementation distinguishes, among other cases, flaky tests, application bugs, stale tests, broken test code or framework, dependency or environment problems, investigation, and waiting-on conditions. Use categories that make sense in your own project; do not label an application regression as mere flakiness.
Choose a framework-supported quarantine mechanism
There is no single syntax that applies to every test framework or CI system. Use the mechanism your repository supports, and verify exactly which jobs it excludes and whether the test remains runnable outside the blocking path. GitLab’s RSpec and Jest conventions are examples from GitLab’s repository, not general framework defaults.
GitLab’s RSpec example
GitLab documents attaching quarantine metadata and an issue URL to an example or enclosing context, with typed metadata such as :flaky or :bug. Follow the current guide for the repository’s exact syntax and prerequisites: GitLab quarantine implementation. The guide notes repository-specific constraints, including that its shown mechanism cannot quarantine shared examples or calls to it_behaves_like or include_examples. It also lists requirements such as feature-category metadata, a test-failure issue, and linking the merge request to that issue. Do not assume these constraints or requirements apply outside GitLab.
GitLab’s Jest example
GitLab’s documented Jest approach marks a test skipped and includes a quarantine comment; the guide also documents a command for running quarantined Jest tests. Use the current repository instructions for the precise comment format and command. Do not copy GitLab’s conventions into another project without checking its test runner and CI configuration.
Make the quarantine temporary and owned
Every quarantine should have an issue, a reason, a named owner, investigation progress, a review date, and a decision about how the test will exit quarantine. GitLab’s handbook offers one concrete policy model: it separates a fast path for urgent blockers from a longer-term path for failures needing more investigation. Its thresholds are GitLab policy, not industry standards.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11| GitLab policy example | Published threshold | What it means in GitLab’s process |
|---|---|---|
| Fast quarantine | 3 days maximum | For a test blocking critical work when the team can update the issue within that short follow-up window. |
| Long-term quarantine | 3 months maximum | For an unknown cause, limited immediate capacity, or investigation expected to take longer than 3 days. |
| Ownership acknowledgement | Within 48 hours | The designated owner acknowledges responsibility and provides investigation and timeline updates. |
| Deletion warning | 1 week | GitLab’s process issues a final warning before its cleanup/deletion process for quarantines reaching the limit. |
These values are from GitLab’s handbook as accessed October 3, 2026; use them as an example of an explicit policy, not as recommended universal deadlines. GitLab’s process also calls for weekly progress updates and resolving or removing long-term quarantines within its stated maximum.
Best Value
Resolve the quarantine and dequarantine deliberately
A quarantine should end with a recorded choice: fix the test and restore it to the blocking path, remove it if it is redundant or cannot be repaired, or replace it with better coverage or a more appropriate test at another level. Do not let an excluded test become an undocumented permanent gap.
GitLab’s handbook requires the root cause to be identified and fixed and gives more than 100 local passes as its stability example before dequarantine, or removal/replacement with better coverage. It also calls for one week of monitoring after reintroduction and immediate re-quarantine if the test fails again. Those are GitLab’s criteria, not a mathematical guarantee or a universal run-count rule; choose a verification plan suited to the failure mode and your CI environment.
Common mistakes and recovery
- A retry passes, so the issue is closed: keep the original failure evidence and investigate the cause. A pass on retry does not show whether the test, environment, or application is responsible.
- The test is skipped with no issue or owner: restore traceability by linking the failure record, reason, owner, and review date to the quarantine.
- A product bug is labeled flaky: reclassify it and route it through the application-bug process; quarantine should not conceal a known regression.
- The test cannot be quarantined by the chosen mechanism: check framework and repository-specific constraints, then use a supported alternative rather than assuming metadata works for every test form.
- The test fails again after reintroduction: preserve the new failure, re-quarantine if your policy calls for it, and reopen investigation rather than treating prior passing runs as proof of permanence.
- The quarantine reaches its deadline without progress: escalate to the owner and make an explicit repair, replacement, or removal decision instead of renewing the exception silently.
Or skip the browser setup
For website screenshots used in test reports, visual checks, or debugging artifacts, ScreenshotNeo is a screenshot API and MCP server. One GET request returns a PNG, JPEG, WebP, or PDF; see the API documentation.
Recommended Free Tools
Quick Recap
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpScreenshotNeo accepts cookie or consent banners and removes more than 60 known consent platforms, newsletter popups, and chat widgets before capture; each step can be turned off. Bot checks/CAPTCHAs, blank pages, timeouts, failed loads, and cache hits cost nothing, and responses indicate the page verdict and billing status. Its MCP server provides take_screenshot, get_page_info, and capture_pdf for AI agents. The Free plan includes 1,000 shots a month with no card; paid plans start at $5 for 3,000 shots, and every feature is available on every plan. Sign up for 1,000 free screenshots a month with no card.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
