Free tools Windows power users keep installed
One-click scans. No signup required.
A green CI run does not prove that the tests you expected actually ran. In Juan Camilo Auriti’s account, a missing test dependency caused pytest to skip a group of tests; the job still succeeded, and the omission went unnoticed for two months. His incident shows why test results need to be read alongside collection counts and skip reports.
How a missing dependency left CI green
Auriti says he added a dependency to a test helper but did not declare it in the project’s development dependencies. It existed in his local environment, so the tests worked there; CI did not have it. The affected tests used aiosqlite = pytest.importorskip("aiosqlite"). When pytest could not import the module, it skipped those tests rather than failing the run.
As an Amazon Associate I earn from qualifying purchases.
In his project, the reported skip count rose from 6 to 168, while CI stayed green for two months. After he declared the dependency and removed importorskip where the dependency was required, he reported 1,452 passing tests. These are figures from Auriti’s first-person account, not independently audited CI results. His account appeared in a DEV Community article whose retrieved passage gives a September 23 publication date but not a year.
The key distinction is between a successful result and a complete test run. As Auriti put it: “The uncomfortable general version: every green build asserts that the tests that ran, passed. None of them assert that the tests ran.” That is his interpretation of the incident, not a formal guarantee about every CI system.
What pytest.importorskip does—and when it fits
Pytest documents pytest.importorskip as importing a module and skipping the current test if the module cannot be imported. That is useful when a test genuinely depends on an optional integration that may not exist in a particular environment. It can also conceal a packaging mistake when the test suite requires that dependency everywhere.
For a required test dependency, declare it in the project’s development or test dependency set and let the run fail if it is unavailable. Reserve importorskip for tests that are intentionally optional; otherwise a missing dependency can turn a broken test setup into an apparently successful build.
Import behavior is version-sensitive. Pytest’s documentation says the traditional behavior of catching ImportError was deprecated in pytest 8.2 and removed in pytest 9.1. Current API documentation describes exc_type as defaulting to ModuleNotFoundError; passing ImportError opts into catching broader import errors. A package that is installed but broken can raise an import error, so check the documentation for the pytest version your CI actually runs rather than assuming every import-time problem becomes a skip.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallChoose checks for the failure you want to catch
Collection and execution expose different problems. A collection floor can detect tests disappearing before execution, but it will not catch tests that are still collected and then skipped. A skip summary makes those skips visible, while a strict skip policy can turn selected skips into failures if that fits the suite.
| Check | What it can reveal | Signal and limitation |
|---|---|---|
| Declare required test dependencies | A dependency missing from the CI environment | Addresses the underlying setup mismatch; it does not report unrelated skipped tests. |
pytest -rs |
Tests skipped during the run and their reasons | Provides a visible summary, but showing skips does not itself fail the build. |
| Minimum collected-test count | A shrinking collection caused by changed paths, selectors, or collection/import problems | Can fail the run when collection falls below a configured floor; does not detect collected tests that later skip. |
| Strict skip-count policy | Skips that exceed a chosen limit | Can provide a hard failure, but legitimate environment-dependent skips may make a threshold noisy. |
| Runtime trend | A sudden change in test duration | A complementary signal suggested in the article’s comments, not a guaranteed detector or benchmark. |
Make skip reasons visible in CI
Run pytest with -rs to include skip reasons in the short summary. Pytest’s output documentation explains the -r summary option and demonstrates skip reasons in its example. This is a low-friction way to make a rising skip count easier to notice in routine logs, but it remains an observation rather than a failure policy.
Use a collection floor for disappearing tests
Pytest exposes the number of collected items as session.testscollected. Auriti demonstrates checking that value in a pytest_sessionfinish hook and failing the run if it drops below a configured minimum, except in collection-only mode. His example uses 1,400 as the threshold; that is an example value from his post, not a general recommendation.
Rank #4
A floor is most useful when the expected test suite is relatively stable and a sudden collection drop should stop CI. Set it from the project’s own normal collection behavior, and update it deliberately when the suite changes. It is not a substitute for inspecting skip output: pytest has already collected tests that later skip, so those tests still count toward the floor.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Apply the checks in the order they diagnose the problem
- Confirm required dependencies are declared. Compare the test helper’s imports with the project’s development or test dependency list and CI install step. If the suite requires a package, CI should install it rather than silently treating it as optional.
- Expose skip reasons. Add
-rsto the pytest command used in CI and inspect the summary for unexpected reasons or changes over time. - Record the collection baseline. Run
pytest --collect-only -qto see how many tests the current configuration discovers. Use the project’s normal result to choose a sensible floor if collection loss is a meaningful risk. - Add a collection guard if appropriate. A
pytest_sessionfinishhook can fail below that floor, with an exception for collection-only runs as in Auriti’s example. - Decide whether skips should sometimes fail CI. If the suite has few legitimate environment-specific skips, a strict skip policy may be practical. If skips are expected across platforms or service availability, review skip reasons and trends instead of using a brittle universal threshold.
Keep the signals separate
A dependency declaration prevents the specific setup omission Auriti described. A skip summary reveals tests that were collected but did not run. A collection floor catches tests that were not collected in the first place. None of these checks replaces the others because they observe different stages of a test run.
Quick Recap
Best Value
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
