Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsOrbi’s reviewer approved two fixes that each introduced a regression because the agents tested the example they expected to work instead of probing the broader input classes the changes could affect. Lawrence Liu’s account of Orbi’s internal benchmark shows why regression testing for coding agents depends on more than asking for tests: the harness needs systematic inputs, an independent oracle, and a benchmark grader known to distinguish broken code from a verified fix.
What the reviewer missed
Orbi’s software harness routes a GitHub issue through implementation, an independent review, and a merge. In the benchmark Liu describes, the reviewer’s verdict on the original fork fixes was “review: pass, no findings.” Both fixes nevertheless introduced regressions.
As an Amazon Associate I earn from qualifying purchases.
A display-width fix broke other Unicode cases
In pyinfra, a change counted wide characters as two columns. It aligned one example, but did not handle other Unicode sequences correctly: an emoji modifier sequence and a decomposed Japanese character behaved incorrectly. Checking the example that motivated the change was not enough to establish correct display width across the relevant inputs.
A secret-replacement fix created new failures
In chainloop, a change trimmed a reported secret before replacing it. That addressed one case but exposed trailing secret characters in another. It also made a one-character secret match too broadly. The problem was not simply a missed test case: both implementation and review had sampled inputs too narrowly to reveal the range of effects.
Liu’s diagnosis is that the agents checked the behavior they expected to work rather than constructing probes across the affected input classes. A passing review therefore reflected the quality of the review process, not proof that the patch was safe.
How Orbi tested the harness
Liu ran Orbi’s delivery loop against private repository snapshots: the system implemented an issue, opened a pull request, ran an independent review, and merged the change. A hidden grader then evaluated the result alongside repository checks and tests.
Before testing harness variants, each grader had to pass a basic calibration: it needed to fail the original code, fail the faulty merged pull request, and pass a hand-verified fix. The benchmark issues were rewritten to remove links, numbers, and hints about the solution. Later runs used wrappers to prevent reviewers from consulting the real upstream answer.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
The first tuning set covered three repositories: pyinfra, chainloop, and fedify. Variants changed prompts, skills, runner tools, implementation models, review models, or combinations of those factors. Liu warns that many model-and-configuration combinations had only two to four runs, and considers the deepseek-flash results the more informative column.
What the benchmark results show
The reported results are evidence about a small, author-run benchmark—not a general performance guarantee. The tuning set had three issues, and the held-out sets had nine ordinary issues and three regression-prone issues. Liu also notes that a hidden grader can detect only defects someone thought to test.
Tuning results improved, but they are not a held-out score
On the three tuning issues, deepseek-flash used in both the implementation and review roles passed 6 of 15 runs with the original harness, compared with 28 of 30 runs for versions v13 through v15. Liu calls this a training score: it measures the issues used to tune the harness, not performance on an independent test set.
Ordinary held-out bugs were solved by all three setups
On nine held-out ordinary issues, each of the three tested setups passed all nine. These were maintainer fixes in Go, TypeScript, and Python; graders used the maintainers’ tests and were calibrated against code from before and after the fix. Liu cautions that these were localized, ordinary bugs that did not require the new regression rules to pass.
Additional regression-prone issues remained difficult
On three further issues where the first upstream fix caused a regression, the original harness passed four scored runs and v15 passed three. The v15 total excludes a completed numpy run because all three attempts were killed for memory. Its three scored runs covered aiohttp once and OpenTelemetry Collector twice.
More checks came with higher resource use
For the nine held-out ordinary issues, v15 used a median 5.8 million tokens per run versus 2.8 million for the original harness, and took 16 minutes versus 11. Liu says roughly 97% of tokens were cache reads; the additional wall-clock time was still real.
In an earlier comparison on nine ordinary unseen issues, v15 used a median 2.0 times the tokens and 1.45 times the wall-clock time. Those ratios describe that comparison; they should not be conflated with the later held-out medians above.
What changed in the harness
Replace self-selected examples with generated input classes
Initially, agents were asked to compare behavior before and after a patch. That left them free to choose their own examples, so they could repeat the same narrow sampling that missed the regressions. Moving the instruction into a skill did not fix the underlying problem.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Later versions required agents to generate inputs programmatically across relevant classes and record how many they checked. For text-formatting changes, examples included ASCII, CJK and fullwidth text, combining marks, emoji modifiers, zero-width sequences, ANSI codes, escapes, quotes, and runs of backslashes. The point is to exercise categories of input systematically, rather than to accumulate a few hand-picked examples.
Rank #4
Make the test oracle independent of the patch
Generated tests can still confirm a faulty assumption if their expected results are derived from the patch itself. Liu describes a run that built a reference formatter by re-implementing the patch’s rule and then reported zero regressions. Agreement between two implementations of the same assumption was not independent evidence that the rule was correct.
A stronger check gets expected behavior from a source independent of the patch: an existing library, known-correct prior behavior, or an invariant. List dependencies before choosing a reference, and compare multiple references when they disagree. If a test oracle merely repeats the patch’s logic, a green result cannot validate that logic.
Probe replacement collisions and use repository conventions
When a patch changes text that is searched for or replaced, check where else that text occurs. The chainloop failure shows why: changing trimming and replacement behavior can affect trailing characters and short secrets beyond the motivating example.
Recommended Free Tools
For project-specific rules, Liu recommends relying on the repository’s own contribution tooling rather than hard-coding assumptions into a general harness. A harness can make checks systematic, but the project’s tooling remains the authority on its conventions.
Best Value
A practical review checklist
- Probe nearby failure modes. After an agent change is merged, construct an input that could expose a related failure and run it—not just the example the patch was written to fix.
- Inspect the oracle. Check how each generated test decides what the correct result should be. If it repeats the patch’s assumption, it is not independent evidence.
- Generate across meaningful classes. Programmatically vary the inputs that the changed behavior can affect, and report how many were checked.
- Use project-native checks. Follow the repository’s own checks and contribution tools to establish project conventions.
- Calibrate the grader and protect the answer. Verify that it rejects known-bad code and accepts a known-good fix, and prevent agents from consulting the answer they are meant to find.
What to take from Orbi’s results
The benchmark supports a specific lesson: systematic regression guards can improve performance on the targeted tuning issues, but they cost time and tokens, and the reported runs do not establish a universal winner. The ordinary held-out results show that all tested setups handled those localized bugs; the smaller regression-prone set shows that the harder cases remain uncertain. Neither result guarantees that a harness will catch defects its grader and generated inputs do not cover.
Liu’s closing advice is worth applying to surprising benchmark outcomes: “When a result surprises you, read the transcript before you believe it.” He also notes that “A benchmark is software, and ours had bugs,” describing grader errors, missing snapshot files, and answer leakage. A benchmark result is only as trustworthy as the process that produced it.
Read Lawrence Liu’s original account at Orbi (September 29, 2026).
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
