A green run tells you one narrow thing: the tests that were configured to run, in that environment, on that execution, did not report a failure. It doesn’t tell you the program is correct. “Nothing” overstates it, but the instinct behind it is sound. If an AI coding agent, a colleague or you wrote the tests, green only means something when the tests check important behavior and report failures reliably.
This article covers what green can and can’t establish, why coverage percentages mislead, how flaky tests damage trust in both red and green, and how to test your tests with mutation testing.
As an Amazon Associate I earn from qualifying purchases.
What a green run does and does not say
| Green tells you | Green does not tell you |
|---|---|
| The tests that ran did not fail on this execution | That the right tests exist for the behavior you care about |
| The code was exercised along some paths | That any path’s output was actually checked |
| The test environment behaved acceptably this time | That production, other platforms or other data will behave the same |
| No regression was caught by the existing assertions | That a regression outside those assertions didn’t happen |
Missing tests, missing scenarios and weak assertions are invisible in a pass. A suite with zero meaningful assertions is also green.
Free tools Windows power users keep installed
One-click scans. No signup required.
Why a green suite from an agent deserves extra suspicion
This is analysis, not a measured finding: when the same author writes the code and the tests, the tests tend to describe what the code does rather than what it should do. An agent that is told “make the tests pass” has an obvious easy path, which is to write or adjust tests that match its implementation. Patterns to watch for in a diff:
- Assertions that only check something exists or doesn’t throw, such as
assert result is not None. - Expected values copied from the implementation’s own output, or computed by the same logic under test.
- Heavy mocking, so the test verifies the mock setup rather than behavior.
- Tests that were loosened, skipped or deleted in the same change that made the suite green.
- Only happy-path inputs: no empty values, boundaries, malformed input or error paths.
The same checks apply to human-written tests. The agent just produces them faster and in greater volume, which makes a skim of “all passing” more tempting.
Coverage shows execution, not detection
Coverage measures which code ran during the tests. It cannot tell whether a wrong result would have been noticed. Google Testing Blog authors Carlos Arguelles, Marko Ivanković and Adam Bender put it directly in Code Coverage Best Practices (2020): “A high code coverage percentage does not guarantee high quality in the test coverage.”
Here is a small example. This function has a bug, since it should apply the discount only at 100 or more:
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
def price(total):
if total > 100:
return total * 0.9
return total
def test_price():
price(150) # runs the discount branch
price(50) # runs the other branch
Both branches execute, so line and branch coverage are 100%, and the test is green. It asserts nothing, and it never tries the boundary at exactly 100.
Google’s coverage guidelines are not a universal target
In the same 2020 article, Google’s general guidelines call 60% “acceptable,” 75% “commendable” and 90% “exemplary.” The authors say there is no ideal percentage for every product. Treat these as one organization’s rough benchmarks. They’re useful for spotting neglected areas, not for certifying a release. Coverage is more informative as a map of what is definitely untested than as a score of what is well tested.
Flaky tests make both colors unreliable
A flaky test passes and fails on unchanged code. That has two costs: a red build may be noise, and a green build may be luck. Teams learn to rerun until green, which trains them to ignore failures.
Rank #4
Google’s figures, reported by John Micco in 2016, show the scale in one very large corpus: 1.5% of test runs were flaky, almost 16% of tests had some flakiness, and about 84% of observed pass-to-fail transitions involved a flaky test. These numbers describe Google’s tests at that time, not software teams in general. Your rate could be much lower or higher, and the way to find out is to track it.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errors- Record results per test over time, not just per build.
- Rerun failing tests on the same commit; a test that changes outcome without a code change is flaky.
- Quarantine flaky tests visibly with an owner and a deadline, rather than silently retrying.
- Look for common causes: shared state, time and date dependence, ordering, real network calls, race conditions and unseeded randomness.
Test your tests with mutation testing
Mutation testing makes small artificial changes to your code, such as flipping > to >= or replacing a return value, and reruns the tests. If the suite still passes, the mutant “survived,” and that points to a missing or weak assertion. In the discount example, changing > to >= survives, which is exactly the boundary gap.
Best Value
The evidence is encouraging but bounded. Petrovic, Fraser, Ivanković and Just analyzed about 15 million mutants at ICSE 2021. They reported that developers who used mutation testing wrote more tests and improved their suites, and that mutants showed evidence of coupling with real faults. That is one industrial dataset. It doesn’t guarantee that a high mutation score means your suite will catch every real bug.
How to use survivors
- Run a mutation tool for your language on a critical module, not the whole repository first.
- Read the surviving mutants and sort them: a real gap, an equivalent change that doesn’t alter behavior, or code that doesn’t matter.
- For each real gap, write an assertion that fails against the mutant and passes against the original.
- Repeat on the modules where a bug would hurt most.
Do this review yourself, or have a reviewer do it. Asking the same agent to kill mutants works as a first pass, but check that the new tests assert behavior rather than reproduce the mutant’s exact output.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.A layered strategy for deciding “enough”
No single number answers “how much testing is enough to release?” A layered approach covers different kinds of risk:
- Fast unit tests for logic, boundaries and error handling, run on every change.
- Integration tests for the seams: databases, queues, external APIs and configuration.
- Critical user-journey checks for the few flows that must never break, such as sign-up, payment and data export.
- Non-functional work where relevant: performance, accessibility, security and privacy.
- Exploratory testing by a person, to find what nobody thought to script.
Keep the suite reliable, maintainable and fast. A slow or flaky suite gets skipped, which defeats the layers above.
A review checklist before you trust a green build
- Did the change add tests for the behavior it introduced, including a failing-first check where practical?
- Does each new test assert a specific expected outcome that came from the requirement, not from the code?
- Were any tests removed, skipped or loosened in the same change?
- Do boundaries, empty inputs and error paths appear?
- Did you run the suite more than once, or in a different order, to expose flakiness?
- Do surviving mutants in the changed code reveal unasserted behavior?
- Has someone run the critical journey in a production-like environment?
The Bottom Line
Treat green as a claim to verify, not a verdict. Ask what the tests assert, how reliably they report, and whether they’d notice a small deliberate break. If you can’t answer, the build is green but you still don’t know.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
