A test that fails once and passes on rerun is not automatically safe to merge. Treat the second result as evidence that the behavior may be intermittent, preserve the first attempt, and investigate the test’s state, fixtures, hooks, and execution environment. A retry can reduce disruption, but it cannot prove that the original failure was harmless.
What a green retry actually tells you
In pytest terminology, a flaky test fails sporadically or non-deterministically. When the same test passes after a retry, you have learned that the failure was not reproduced under the second attempt’s conditions. You have not learned why it failed, whether the failure could recur for another commit, or whether the first result exposed a genuine product defect.
As an Amazon Associate I earn from qualifying purchases.
That distinction matters because CI is a merge signal. If developers learn that a red job can be made green by clicking retry, they may begin overlooking real failures. Pytest’s documentation warns that unreliable results reduce trust in the suite, can cause genuine failures to be missed, and consume time through repeated reruns and investigations.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Selective documents one operational model in which a pass after retry is recorded as a flake occurrence. That classification is useful for reporting, but it is a vendor’s model rather than an industry-wide standard. Your team should define its own event names and metrics explicitly.
The failure lifecycle: preserve first, retry second
1. Capture the first attempt before doing anything else
The first attempt contains the most valuable diagnostic context. Retain the assertion output, stack trace, test order, random seed, environment variables relevant to the test, dependency versions, timestamps, worker or shard identity, and the CI job’s logs and artifacts when those are available.
Do not overwrite the original report with the retry report. Store them as separate attempts linked to the same test execution. If the runner only emits one final status, add an external record or artifact so the initial failure remains visible to the person reviewing the merge.
- Record the commit, branch, runner image, operating-system version, and Python and pytest versions.
- Keep database, service, browser, container, and network logs created during the failed attempt.
- Preserve ordering and seed information so another engineer can replay the same schedule.
- Mark whether setup, the test call, or teardown produced the failure.
2. Retry to gather evidence, not to erase the result
A retry can distinguish a repeatable failure from an intermittent one. If the same assertion fails again, treat it as a normal failure and investigate the product or test. If it passes, label the result as “passed on retry” or an equivalent state and retain both outcomes.
Pytest describes reruns as mitigation, not a root-cause fix. A rerun can keep a transient infrastructure problem from blocking every pipeline, but it does not repair shared state, timing assumptions, or an incorrect assertion. Make the retry count and decision rule visible in CI configuration rather than hiding them in an opaque wrapper.
3. Compare the two attempts
Look for differences that could explain the outcome: worker assignment, test order, service readiness, clock time, available CPU or memory, network responses, random seed, and fixture data. A retry that runs in a different process or after another test has cleaned up may be testing a different state, not proving that the first failure was accidental.
Where hooks fit—and where they do not
“Hooks” are extension points in a test runner or CI system. The sources for this topic do not establish a universal hook API or a canonical sequence, so treat concrete implementations as framework- and version-specific. Verify names and behavior against the versions your repository runs.
In practice, teams use hooks or equivalent callbacks at several boundaries:
- Attempt start: assign an attempt identifier and capture commit, worker, seed, and environment metadata.
- Test outcome: record setup, call, and teardown results independently.
- Retry decision: apply the documented limit and classify a second outcome without deleting the first.
- Artifact collection: attach logs, screenshots, dumps, and fixture diagnostics to the failed attempt.
- Quarantine handling: mark a test as non-gating while continuing to execute and report it.
- Pipeline gate: evaluate the team’s chosen budget and return a merge-blocking status when policy requires it.
A hook should add observability and enforce policy; it should not silently convert an assertion failure into success. Keep policy decisions in configuration that reviewers can inspect, and emit a machine-readable record containing every attempt.
Fixtures are frequent sources of intermittent state
Pytest identifies inadequate isolation, ordering dependencies, uncleaned data, interactions during parallel execution, global state, timing-sensitive assertions, and thread-safety problems as possible causes of flaky tests. Fixtures are natural places to inspect because they create, share, and destroy much of that state.
Check setup and teardown symmetry
For every resource a fixture creates, identify the corresponding cleanup path: files, database rows, schemas, ports, processes, threads, subscriptions, temporary credentials, and environment-variable changes. Cleanup must run when the test fails, not only when it passes. A fixture that yields a resource and performs cleanup in a guaranteed finalization block is easier to audit than one that relies on a later test to reset state.
import pytest
@pytest.fixture
def isolated_store(tmp_path):
store = create_store(tmp_path)
try:
yield store
finally:
stop_background_workers(store)
close_store(store)
This is an illustrative pattern, not a drop-in hook recipe. The cleanup functions, fixture scope, and concurrency behavior must match your application.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteAudit fixture scope
A function-scoped fixture normally gives each test a fresh instance, while broader scopes can improve speed by sharing state. Sharing is safe only when the resource is deliberately immutable or reset completely between users. A session-scoped database, cached client, or global configuration can let one test influence another through data, connection pools, environment variables, or in-memory caches.
Look for teardown that is too late or incomplete
Failures during setup can prevent ordinary cleanup code from running. Teardown can also fail after the assertion has passed, leaving a contaminated environment for later tests. Report setup, call, and teardown separately so a green test body does not conceal a lifecycle failure.
Inspect concurrency and background work
Tests and fixtures may spawn threads or processes. A worker that outlives its fixture can write to a file, database, queue, or port while another test is running. Join threads, stop consumers, close sockets, and wait for subprocesses to exit before releasing the fixture. If parallel workers share an external resource, partition it by worker or make access synchronization explicit.
Common causes of a pass-on-retry result
- Ordering dependence: the first test expects state left by an earlier test, or the retry happens after that state has changed.
- Timing sensitivity: a fixed sleep is shorter than service startup, replication, event delivery, or eventual consistency.
- Uncontrolled global state: environment variables, module caches, singleton objects, locale, timezone, or current working directory are not restored.
- Resource leakage: files, rows, ports, threads, or processes survive teardown.
- Parallel interference: workers use the same account, queue, port, directory, or database records.
- External dependency variability: a network service, clock, DNS response, or rate limit changes between attempts.
- Thread-safety defects: shared objects are read or mutated without the synchronization the test assumes.
- Weak assertions: the test checks a transient value or an overly narrow timing window instead of the contract.
Randomizing order and replaying a recorded seed can expose dependencies, but neither technique fixes the underlying isolation problem. Use them to make a failure reproducible and then repair the test or system boundary.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Rank #4
Choose a response: fix, redesign, or quarantine
Fix or rewrite the test
Prefer a deterministic test that controls its data, waits on a meaningful condition, and cleans up every resource. If an end-to-end test is carrying too much state, move part of the behavior to a unit or integration test with a narrower boundary, while retaining a smaller number of end-to-end checks for the contract that truly needs them.
Use retries as a controlled mitigation
Retries are reasonable when an independent transient failure is understood and the first result remains visible. Set a finite limit, record each attempt, and alert on repeated retry passes. A policy that retries indefinitely merely hides a broken gate.
Quarantine with an exit condition
Selective describes quarantine as continuing to run and report a test while preventing it from failing the run. That can protect delivery while an owner works on the defect, but it must be temporary and visible. Pytest cautions that permanent, non-strict xfail usage can become manual quarantine and is dangerous when left in place indefinitely.
Attach the following fields to every quarantined test:
- A named owner or team.
- The issue or change that introduced quarantine.
- The observed symptom and the evidence retained.
- A review date or trigger, such as the next release or a fixed number of runs.
- The restoration condition: a merged fix, a rewritten test, or an explicit decision to delete the test.
Designing a flake budget without inventing a universal percentage
No universal flake-budget threshold is established by pytest documentation, GitLab engineering guidance, or Selective’s vendor documentation. A budget is a measurable agreement between test owners and the people who rely on the merge gate. Choose values that reflect your risk, test volume, release cadence, and ability to investigate.
Best Value
Write the policy before choosing a number. At minimum, define these dimensions:
| Dimension | Decision to record | Why it matters |
|---|---|---|
| Unit | Individual test, suite, job, or complete pipeline | A suite-level rate can hide one repeatedly failing test. |
| Event | First-attempt failure, pass after retry, quarantine result, or a stated combination | Different events imply different urgency and must not be combined silently. |
| Denominator | Test executions, jobs, or CI runs | A rate is uninterpretable without knowing what was counted. |
| Window | Calendar period, release, or rolling number of runs | A short outage and a long-term trend require different responses. |
| Evidence | Logs, seeds, environment data, and attempt-level status | Investigators need enough context to reproduce the problem. |
| Response | Owner assignment, escalation, merge block, or quarantine review | A budget without a consequence is only a dashboard metric. |
| Quarantine lifecycle | Who reviews it and when it returns to the gate | Temporary controls otherwise become permanent suppression. |
For example, a team might count every first-attempt failure and every pass-after-retry separately over a rolling period. That is a policy choice, not a standard recommendation. The important property is that anyone reviewing a merge can see the event definition, denominator, window, and action tied to the measured value.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Comparing operational approaches
| Approach | Questions to ask | Main caution |
|---|---|---|
| Test-runner retry or diagnostic plugin | Are all attempts exposed? Can a failure be replayed? Can order be randomized? Is it compatible with the framework version? | Rerunning may reduce disruption without fixing the test. |
| Quarantine | Does the test continue running? Is it still reported? Are ownership and review status tracked? | Permanent suppression can hide failures; non-strict xfail should not become an indefinite workaround. |
| CI analytics and reporting | Does it retain history and occurrence context? Does it distinguish retry passes from repeated failures? Does it integrate with your repository and runner? | Definitions and features differ by product, so validate the behavior rather than assuming a label means the same thing everywhere. |
A practical implementation sequence
- Define the status vocabulary. Use distinct states for first-attempt failure, retry pass, repeated failure, quarantined result, and infrastructure failure.
- Preserve attempt artifacts. Configure the runner or CI layer to retain the original log and metadata before launching a retry.
- Set a finite retry rule. Document which failures are eligible, how many retries occur, and which result controls the merge gate.
- Instrument fixture boundaries. Log resource creation and cleanup, fixture scope, worker identity, and teardown failures without exposing secrets.
- Review order and parallelism. Re-run with a recorded seed, vary execution order, and isolate shared resources by worker.
- Assign ownership. Every confirmed flake receives an owner, an issue, and a next investigation step.
- Apply quarantine only with a deadline. Keep the test running and reported, and schedule a restoration review.
- Measure the chosen budget. Publish the numerator, denominator, window, and action when the limit is exceeded.
Troubleshooting by symptom
The test fails only when the full suite runs
Run it after different preceding tests and with randomized order. Inspect global variables, caches, environment changes, database rows, temporary directories, and ports that the preceding test may leave behind. Convert hidden ordering assumptions into explicit setup.
The test fails mostly on parallel workers
Check for shared usernames, directories, queues, ports, and database records. Include the worker identifier in resource names or allocate isolated resources. Look for background threads and subprocesses that continue after teardown.
The retry passes after a fixed delay
Replace arbitrary sleeps with polling for a specific observable condition and a bounded timeout. Capture service readiness, event delivery, and clock values so the failing timing window can be identified.
The test is green but teardown reports errors
Treat teardown as part of the test’s reliability contract. Make cleanup failures visible, preserve their logs, and repair the fixture before allowing the test to be considered stable.
The same test passes on one runner image and fails on another
Compare dependency versions, operating-system behavior, locale, timezone, filesystem semantics, CPU and memory limits, and service configuration. Record the environment with each attempt instead of treating the runner label as sufficient context.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWhat a trustworthy merge signal looks like
A trustworthy gate does not require every transient event to block every developer, but it does make uncertainty visible. The first failure remains available, the retry result is classified rather than substituted, fixture and hook behavior can be inspected, and quarantine has an owner and an end date. When the team’s chosen budget is exceeded, the policy—not an individual’s habit—determines whether to stop merging, escalate investigation, or revise the test.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
