Three AI models scored perfectly on the first version of an outage-response benchmark, which showed that the test was too easy to tell them apart. After Jared Chu changed one observation at a time in matched scenario pairs, two models still scored 36 of 36 and one scored 33 of 36. Chu published the work on September 24, 2026 as a Kaggle Benchmarking Challenge submission. These are his benchmark results, drawn from short fictional multiple-choice cases. They have not been independently validated.
What the first version measured, and why perfect scores were a warning
The pilot posed five fictional incidents covering DNS, TLS, deployment rollback, backup recovery, and an incomplete outage report. For each one, a model selected an action and a supporting evidence statement, then wrote a short explanation. Each incident was shown in three shuffled answer orders, which gave 15 responses per model. All three models scored 15 of 15.
As an Amazon Associate I earn from qualifying purchases.
A perfect score on that set meant the cases could not separate the models. It did not show that the models handled outages equally well. The pilot could not distinguish a model that noticed what the evidence did and did not prove from one that happened to pick the right answer.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Changing one observation at a time
The follow-up used six matched pairs. Within each pair, the incident description and the available choices stayed the same while a single observation changed. Because that observation changed, both the keyed action and the keyed evidence statement changed too. Chu frames the underlying question as: given the evidence and permissions in an incident report, what should happen next? The pairs test whether a missing observation or a missing permission should alter the action.
#1 Best Overall
The six pairs cover these situations:
- Whether a prior image passed a compatibility test against the current database schema.
- Whether DNS tests isolated DNSSEC validation.
- Cached errors versus origin errors.
- Backup validation.
- DNS-change approval.
- Queued-job durability.
Each variant ran in a fresh conversation. Matched variants used the same option positions across three shuffled orders, with order seeds 11, 29, and 47. That produced 36 responses per model. Those 36 responses come from six authored pairs, not 36 independent incidents, and should be read that way.
The cases and the deterministic scorer were frozen before the follow-up calls. The follow-up was still shaped by the pilot’s ceiling, so it is not an untouched holdout set.
How responses were scored and run
- A response earned one point only if it followed the exact JSON schema and selected both keyed choices.
- The explanation was kept but not judged automatically.
- An infrastructure error would invalidate a run rather than count as a wrong answer.
- All 108 follow-up responses were kept, matched to their frozen prompts, and rescored locally. The aggregates matched Kaggle’s task results.
The three models were chosen before the pilot, with one available model from each of three providers. Chu does not claim that each is that provider’s strongest offering. All runs took place on September 24, 2026 using Kaggle platform defaults, with no sampling overrides. The pilot, saved task reruns, and follow-up are separate result sets and were not pooled.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Results
All figures below are the author’s benchmark results for these runs, not published population statistics.
Rank #3
| Model | Pilot (15 responses) | Follow-up joint score (36 responses) | Both variants correct, per pair and order (18 units) | All three orders correct, per pair (6 pairs) |
| Gemini 3.7 Flash | 15/15 | 36/36 | 18/18 | 6/6 |
| GPT-5.4 mini | 15/15 | 36/36 | 18/18 | 6/6 |
| Claude Haiku 4.5 | 15/15 | 33/36 (91.7%) | 15/18 | 3/6 |
The joint score counts each variant separately. The stricter measures require both variants of a pair to pass in the same order, or all three orders to pass for a pair. Read the headline percentage together with these measures, because a single score hides where the misses sit.
Where Claude Haiku 4.5’s three misses came from
Haiku’s three misses fell in three different pairs, and they were not the same kind of error. Two were disagreements about the keyed action. The third was a schema violation. Treating them as one failure rate would obscure that difference.
DNS: right diagnosis, different priority
The model correctly recognized that the test had not isolated DNSSEC validation. It then chose to prioritize DS and DNSKEY inspection over the broader resolution trace the key called for, so its action did not match the key.
Free tools Windows power users keep installed
One-click scans. No signup required.
Cache: additional diagnostic before eviction
The model recognized successful cache-bypass evidence but chose to inspect origin health before cache eviction. Chu says an extra diagnostic like this may be defensible. He also notes that this result does not establish unsafe behavior.
Durable queue: right IDs, invalid format
The model selected both keyed IDs but added an unrequested reason2 field to its JSON, which violated the exact schema. Under the strict scorer this counts as a miss even though both choices were correct.
What the scores do not show
- These were short multiple-choice exercises with explicit runbooks and some easy distractors.
- The models did not investigate a live outage, execute a change, handle new evidence over time, or demonstrate recovery.
- The evidence choices measured recognition of appropriately scoped claims. They did not measure general confidence calibration.
- Six authored pairs cannot establish a general ranking of models.
- The shuffled orders reveal variability, but prompts were not repeated enough to separate option-position effects from sampling variability.
- No independent expert validation or human manual review was carried out. AI tools drafted the cases, implemented and ran the evaluation, analyzed the outputs, and helped write the article. This limits how far the answer key can be treated as operational ground truth.
- All cases are fictional, with no customer data and no real infrastructure changes.
Inspecting and reproducing the benchmark
The Kaggle project has separate tasks for the pilot and for the paired follow-up. The paired notebook publishes the full corpus, the answer key, the scorer, and the run exports. To compare all three models’ traces, open the registered paired task and use its Compare Outputs view. The 0.00 model headers Kaggle displays reflect a “No overall score” setting. They are not additional measured results.
Quick Recap
- Task registration and model execution come from the Kaggle Benchmarks SDK. The case content and scoring logic were written for this submission.
- The work is released under the Apache 2.0 license.
- The frozen paired corpus and scorer have SHA-256 hash 0def44fe0c0e9d483487ecaaa0b8a8ccba4a30c8127b02e11e3a91d1eab34295.
- Platform views may change, so if the interface has moved, use the notebook’s exported run files as the reference.
Applying the method to your own runbooks
- Start with a case where the correct action is clear and the model passes. Keep the incident text and the options fixed.
- Change one fact, such as turning a failed compatibility check into a passed one. Update the keyed action and the keyed evidence statement together.
- Run each variant in a fresh conversation, keeping option positions matched across several shuffled orders.
- Score schema validity separately from the choice of action, so a formatting slip is not mistaken for a wrong decision.
- Have an operator review any disputed key before treating it as ground truth.
“
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Recommended Free Tools
