Recommended Free Tools
A green build tells you the test runner did not report a failure. It does not tell you that any tests ran. Those are different facts, and a pipeline can produce the first while the second is false. Checking a green result means verifying two things separately: that the expected tests were collected and executed, and that those tests can fail when behavior is wrong.
Two kinds of green
In pytest, the exit status is the signal CI should act on. According to pytest’s current stable documentation on exit codes, exit code 0 means tests were collected and passed, and exit code 5 means no tests were collected. Those two outcomes can both look like a quiet pipeline, but only one of them means work was done.
As an Amazon Associate I earn from qualifying purchases.
| Outcome | Exit code | What it establishes | What it does not establish |
|---|---|---|---|
| Tests collected and all passed | 0 | At least some tests ran and passed | Whether the intended tests were the ones collected |
| No tests collected | 5 | Nothing ran | Anything about the code |
| Runner status hidden by a wrapper | Not visible to CI | Only that the wrapper returned success | Whether pytest collected, ran, or passed anything |
The third row is the one that causes most silent failures. A step such as pytest tests/ || true, a job marked continue-on-error, or a pipeline that pipes output through another command can all turn an exit code of 5 into a passing step. Check the exact command your CI job runs and confirm that a run with zero collected tests fails the job.
How a suite quietly shrinks to nothing
Pytest finds tests through discovery. By default, it looks for files and functions that follow its naming conventions, and discovery can be changed through configuration. Because of this, a change that looks unrelated to testing can change what runs. The essay that prompted this article gives a renamed-directory example. The official discovery documentation describes the default rules and the configuration options that override them.
The common causes fall into a few groups:
- Path changes. A directory is renamed or moved, so the path passed to pytest, or the path in a CI step, points at a location that no longer holds tests.
- Naming drift. Test files or functions are renamed to names that fall outside the default patterns, or a custom discovery pattern in configuration no longer matches the files.
- Ignore and filter settings. An ignore rule, an option in a configuration file, or a
-kor-mexpression in a shared command or environment variable excludes tests. - Invocation differences. A test run started from a different working directory, or with a different path argument, collects a different set of tests than the one you reviewed.
Collection, execution, and skipping are three separate stages. A test can be collected but skipped, or collected and deselected by a filter. A count that stays flat can hide a jump in skipped tests, so check the execution summary as well as the collected total.
Establish a baseline count
The essay’s central practice is to watch the test count. Start by recording what a known-good commit collects.
- On a commit where you are confident the suite is complete, run
pytest --collect-only -qfrom the project root, the same way CI runs tests. - Save the list of collected test IDs to a file in the repository, for example
tests/collected_baseline.txt. The-qoutput lists one test ID per line, so the file can be compared line by line later. - Record the total count in a separate file, for example
tests/expected_count.txt, so CI can compare a single number. - Add a CI step that runs collection and fails if the count differs from the recorded value.
A simple version of that step looks like this:
set -eo pipefail
pytest --collect-only -q | tee collected.txt
actual=$(grep -oE '^[0-9]+ tests? collected' collected.txt | grep -oE '^[0-9]+')
expected=$(cat tests/expected_count.txt)
[ "$actual" = "$expected" ] || { echo "collected $actual, expected $expected"; exit 1; }
The summary line format can vary between pytest versions, so confirm the regular expression against your own output before relying on it. When tests are added or removed on purpose, update the baseline in the same commit so the change is reviewed rather than discovered later.
Free tools Windows power users keep installed
One-click scans. No signup required.
Investigating an unexpected drop
When the count falls without a deliberate reason, do not accept the new number. Work through these checks in order:
- Diff the current list of test IDs against the saved baseline. The diff shows exactly which tests disappeared, which is more useful than the total.
- Confirm that every directory holding tests still exists at the path the CI step uses.
- Compare test file and function names against the discovery patterns in your configuration.
- Check ignore rules,
-kand-mexpressions, and any options in shared configuration or environment variables. - Search for newly added skip markers. Skipped tests are collected but do not execute, so they can make a green run look more complete than it is.
- Read the collection output for errors. A module that fails to import can prevent its tests from being run, and the failure appears in the output rather than as a pass.
Show that the tests can fail
The essay’s second practice is to break the code on purpose and confirm that the suite notices. The point is to test the tests: a test that has never failed has not demonstrated that it can catch anything.
- Work on a branch or in a throwaway clone, never on a shared branch.
- Choose a behavior that matters, such as a boundary condition, a branch in a decision function, or a returned value that callers depend on.
- Introduce a small, representative defect: invert a comparison, return a fixed wrong value, or remove a check.
- Run the tests that cover that behavior and confirm that at least one fails. Note which test failed and why.
- If nothing fails, the suite does not detect that defect. Add or strengthen a test for that behavior before moving on.
- Revert the change and confirm the suite is green again.
Treat this as a diagnostic exercise. A single defect shows that one class of error is caught in one place. Rotating the chosen behavior across modules over time gives a better picture than a single check, but none of these checks proves that all production behavior is covered.
Rank #4
Why coverage does not settle the question
Coverage reports describe which lines executed during the run. They do not compare the run against the list of tests you intended to execute, and they cannot tell whether a test asserted anything meaningful. The essay makes this point as a caution: a coverage figure can look healthy while the intended tests were never collected, so coverage should not be read as evidence that collection worked. Keep the count check and the mutation check as separate steps alongside any coverage reporting.
Assertions that cannot fail
A test with a weak assertion can stay green through a defect. Tests that only confirm a call completed, or that a value exists, do not distinguish correct behavior from incorrect behavior. Compare these two versions:
Best Value
# Passes for almost any output
def test_discount():
assert apply_discount(100, 10) is not None
# Fails if the calculation is wrong
def test_discount():
assert apply_discount(100, 10) == 90
The first test would still pass if the function returned 0, 1, or a negative number. The second one fails for any incorrect result. When reviewing a suite, look for assertions that check existence, type, or absence of an exception without checking the value the behavior is supposed to produce.
“A test you have never seen fail has told you nothing so far.” (Serguey Asael Shinder, DEV Community essay, dated only as “Sep 16” on the page)
The essay’s advice applies to any runner, but the exit-code and collection details above are specific to pytest. Other frameworks have their own discovery rules and exit behavior, so the same three questions apply there with different commands.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →The Bottom Line
A green pytest run proves that something was collected and passed, not that the intended tests ran or that they would catch a defect. Verify the collected count against a recorded baseline, make CI fail on an empty collection, and break a behavior on purpose from time to time to confirm a test fails.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
