No—the available results do not show that Span-01 and Mercury Decide earned the same score or failed in opposite ways. Respan reports Span-01 performance on behavior-classification benchmarks; a Reddit author reports Mercury Decide performance on a narrow Korean-language test of Roblox Terms of Service reports. Span-01 was not tested on those same cases, so the numbers cannot establish a head-to-head result.
What each system is designed to do
Span-01: classify behaviors in conversation traces
Respan describes Span-01 as a classifier that applies natural-language behavior definitions to conversational traces. For each behavior, it returns probabilities for present, absent, and not_observable in one forward pass. Developers can combine those scores with thresholds and code to alert, block, log, route cases to a human, or send uncertain results for review. Respan’s explanation is in its September 24, 2026 launch post; implementation details are in the Span-01 documentation.
As an Amazon Associate I earn from qualifying purchases.
Mercury Decide: structured decision questions
Mercury Decide is described as a decision model for Choice, Score, and yes/no questions, with probabilities in its output. The profile describes access through OpenRouter’s System One endpoint and labels the service early access. Its JevBench ranking and claim of up to 14 decisions per second are attributed to Inception in that profile, not independently verified there. These details are presented in a Mercury Decide profile dated October 1, 2026.
What the reported numbers actually measure
| System and result | Test scope | What the figure can tell you |
|---|---|---|
| Span-01: 0.843 overall F1 | Respan’s behavior benchmark; the overall score is the unweighted mean of English and multilingual F1, according to Respan. | Performance on Respan’s benchmark—not accuracy on Korean Roblox reports. |
| Span-01: 0.806 overall F1 | Respan’s production behavior benchmark. Respan’s same table reports 0.716 for Jev, 0.719 for Sonnet 5, and 0.885 for GPT-6 Sol. | A result within that production benchmark; it is not a Mercury Decide score. |
| Mercury Decide: 66.7% accuracy; 28 false negatives out of 90 cases | A Reddit benchmark author’s Korean-focused test of whether chat logs violate Roblox Terms of Service, dated October 1, 2026. | An author-reported result for one task and test setup, not a general model ranking. |
The Span-01 figures are published by Respan in its launch post and documentation. The Mercury Decide result and its stated test limits appear in the Reddit benchmark post. F1 and accuracy are different metrics, and neither the task nor the test cases are shared here; comparing the values as if they were scores on one contest would be misleading.
#1 Best Overall
Why the apparent failure patterns are not opposites
The Reddit author reports 28 false negatives among 90 Korean-language cases and says Mercury Decide seemed to answer “no” on almost every possible report case at the tested threshold. That points to a potential problem for the author’s specific reporting workflow: missed cases. It does not establish how Mercury Decide performs on other languages, thresholds, or decision tasks.
Respan’s Span-01 material covers behavior detection across English and multilingual data, including jailbreak and prompt injection, safety and refusals, privacy and secrets, hallucination and grounding, agent and tool reliability, task and instruction following, and response quality. Those categories are not the same as deciding whether a Korean chat log violates Roblox’s rules. The Reddit post does not report Span-01 results on its cases, so there is no evidence here that Span-01 made the opposite errors—or any comparison of the two systems’ false-positive and false-negative rates on a common test.
There is also a qualification on the vendor benchmark: ModelSystem.One notes that Respan’s benchmark labels are model-generated rather than ground truth, produced mostly through agreement between GPT-5.6 Sol and Claude Opus 5. That matters when interpreting the published Span-01 figures; they should be understood as results against that labeling process, not as a definitive measurement against independently established truth. See the ModelSystem.One profile for its benchmark-label caveat.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →What a fair Span-01 vs Mercury Decide test needs
A defensible head-to-head would run both systems on the same cases with the same target behavior, labels, and decision threshold. For a report-review workflow, the test should also state whether the objective is to catch every reportable case or to avoid false accusations; those priorities can favor different thresholds.
- Task and output fit: define whether the job is behavior-probability monitoring across traces or a fixed Choice, Score, or yes/no decision.
- Error balance: publish confusion counts, including false positives and false negatives, at the same predeclared threshold. Accuracy or F1 alone can hide an error pattern important to the workflow.
- Consistency and adversarial behavior: test whether equivalent inputs change the outcome and whether prompt injection affects decisions. Respan includes paired flip rate and injection attack success rate in a separate evaluation of 11 decision models; that is not a Mercury Decide comparison.
- Calibration: if a system returns probabilities, compare those probabilities with observed outcomes on the same labeled set and name the calibration metric. Respan reports expected calibration error in its separate decision-model evaluation.
- Language and coverage: report results by language and use case rather than generalizing the Korean Roblox test to broad performance.
- Reproducibility and access: identify model version, endpoint or access route, test date, case construction, label process, and any limits that affect results.
Respan reports that its evaluation covered 11 decision models across accuracy, consistency, injection resistance, and calibration. For Jev 1.13.0 it gives 0.932 accuracy, 0.021 paired flip rate, 0.063 injection attack success rate, and 0.045 expected calibration error. This is a vendor-published evaluation in which Span-01 supplies the evaluation signal, not a matched Mercury Decide test; the results and methodology are described in Respan’s launch post.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Access and operational details are separate from benchmark quality
Respan lists Span-01 input pricing and free output in its documentation. The Mercury Decide profile describes a free early-access route, while noting that some usage limits and paid pricing are unpublished. These terms can change, so check the providers’ current access details before planning a deployment; they do not make the benchmark results more comparable.
Quick Recap
Best Value
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
