October DealsAmazon USOctober deal check: compare before you payAmazon US: current deals, useful picks and tech finds.Check DealsWindows FixRecommendedWindows errors stealing your time? Find the fix fastScan stability, cleanup and performance issues.Fix NowOctober DealsAmazon USDeal season is back - check today's better picksAmazon US: current deals, useful picks and tech finds.See Picks×
Skip to content
World desk4 min

181 Code Changes, One Test: Did the Regression Tests Notice?

Receipts tested whether changed tests failed when source changes were reverted. Most judged changes in the selected samples had at least one test that noticed the change, but the study does not prove fixes were correct or represent the wider software ecosystem.
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

In a 2026 study of 181 changes across 17 open-source projects, most judged changes had at least one test that failed when the changed source code was reverted. That is evidence the test noticed the change—not proof that the fix was correct or that the result applies to all software projects.

What the 181-change study measured

Receipts, an open-source tool, runs tests associated with a code change twice: first with the change applied, then after reverting the changed source files to their parent-commit or pull-request merge-base versions. The tests, dependencies, and configuration stay at their newer versions in both runs. The question is whether a test that passes with the change fails against the older source.

As an Amazon Associate I earn from qualifying purchases.

The study, published by syntaxixr in 2026, covered two samples: 81 maintainer fix commits drawn from 12 libraries, and 100 agent-fingerprinted pull requests drawn from five agent-heavy repositories. Maintainer commits were selected from recent qualifying changes, up to eight per repository; the agent sample included up to 20 of the newest qualifying PRs per repository. Agent PRs could be open or closed, and some were unmerged.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

This is a counterfactual check of test sensitivity. It asks whether a test reacts to the source change, not whether the change fixes the underlying bug, covers every relevant behavior, or is safe to ship. The study describes its samples as small and non-random, so the results should not be read as ecosystem-wide rates. Study methods and results · Receipts project documentation

What the study found

Ten maintainer changes and nine agent PRs could not be judged for environmental reasons. The percentages below use only the remaining judged changes.

Sample Judged Proven Weak-only
Maintainer fix commits 71 of 81 64 of 71 (90%)
Agent-fingerprinted pull requests 91 of 100 75 of 91 (82%) 9 of 91 (10%)

“Proven” means at least one test failed against the old source, with no weak or theater result. The figures show that many changes in both samples had tests sensitive to the source change. They do not establish that maintainer-authored tests are generally better than agent-authored tests: the repositories and selection methods differ, PRs included unmerged work, and agent attribution relied on fingerprints rather than verified authorship.

The agent sample included 87 PRs with Claude Code fingerprints, six with Codex fingerprints, six with Cursor fingerprints, and one with a Copilot fingerprint. Those counts describe detected fingerprints in this sample, not exclusive authorship; the study notes that people may have steered the work.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Why a failing test can still be weak evidence

Nine of the 91 judged agent PRs were classified as weak-only. In the described pattern, a test module imports a name introduced by the change at the module’s top level. When the changed source is reverted, that name no longer exists, so the test module fails to load. The failure looks like a test noticing the change, but the test never reaches the old behavior it was meant to check.

A practical mitigation is to import the new name inside only the tests that need it, rather than at module load time. That lets the test module load against the old source, making it possible to distinguish a genuine behavioral failure from a missing import. The study discusses this pattern in an example from a Claude Agent SDK Python PR. The article’s example and explanation

Receipts uses several verdicts to distinguish outcomes:

  • PROVEN: A test fails without the change.
  • GUARD: A test passes on both sides but sits alongside a test that proves the change.
  • THEATER: Tests pass with and without the change, and none proves it.
  • WEAK: A test fails against old source because code it calls did not exist yet, rather than because it detects changed behavior.
  • BROKEN, FLAKY, and SKIPPED: Other outcomes the study tracks, including tests that cannot yield a reliable pass/fail comparison.

At the change level, the study calls a change “mixed” when some tests prove it while others are weak; “unproven” when every test passes without the change; and “weak only” when the tests fail against old source solely because the called code is new.

Free tools Windows power users keep installed

One-click scans. No signup required.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Why some changes are hard to prove with a test run

The study says theater was uncommon in its samples, but gives examples that show why a green or unchanged result can be ambiguous: a type-only change may not affect runtime tests; a Windows-specific newline fix may not be exercised on Linux; a dateutil representation fix may produce output that already matches inherited behavior; or a maintenance commit may mention an issue without changing behavior that the selected tests can demonstrate. A test verdict depends on both the nature of the change and the environment where it runs.

How to use this idea on your own changes

A normal green test run tells you that the tests passed with your code. A revert comparison asks a different question: would those same tests fail if the source change were absent? That can expose tests that pass but do not detect the change, while import-time failures can reveal cases where an apparent failure is not behavioral evidence.

Receipts’ README describes a CLI, a GitHub Action, and an agent skill, and lists support for pytest, vitest, and jest. It states that the tool is deterministic, needs no LLM or API keys, and runs a project’s own test runner. The documented baseline is Node 20+ and Git. The Action can report on pull requests and fail checks for configured verdicts; its example recommends the pull_request event and checkout credentials that do not persist. A report comment requires comment permission. These are capabilities described by the project, not independently verified results. Receipts README

The study also provides reproduction commands, raw repository results, and a Hugging Face dataset. Those materials are the project’s supplied route to inspect or reproduce its analysis; the complete study was not independently rerun here. Study reproduction information

What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

Leave a Reply

Your email address will not be published. Required fields are marked *

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

More from the Wire

  1. World desk4 min
    How to Spot an AI Voice Scam Before Sending MoneyDon’t rely on how a caller sounds. Pause, call back through a known number, and verify the emergency with another trusted person before sending money.
  2. Mountain View desk4 min
    Google’s SynthID Detector: How to Check AI-Generated Images, Video and AudioGoogle’s SynthID Detector looks for an embedded watermark in supported images, video and audio. Here is what its results do—and do not—show.
  3. Redmond desk20 min
    How to create a link to File or Folder in Windows 11Windows 11 gives you several ways to point to a file or folder without moving or duplicating it. You can create a desktop shortcut,…
Recommended PC Tool
Recommended PC Tool
Crashes, No Sound, or Screen Glitches?Free driver scan
Windows Errors? Fix Them Before They SpreadFree repair scan

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.