What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
There is no universal yes-or-no answer: an AI can resist corrections to its own earlier claim, defer too readily to a user, or accept weak evidence simply because it is repeated. The Source-of-Belief Asymmetry Benchmark (SoBA) is designed to test whether a language model changes its answer differently depending on whether the contradicted claim came from the model itself, a user, a document, or no stated source.
Rajan Mishra described SoBA in a DEV Community article published September 25, 2026. Its reported results are the author’s benchmark findings, not independently replicated measurements or evidence about any named commercial AI model.
As an Amazon Associate I earn from qualifying purchases.
What SoBA is designed to find out
SoBA asks whether a model evaluates counter-evidence consistently when the same initial belief is attributed to different sources. In practical terms, it probes two opposing risks: sticking with an earlier answer despite strong contrary evidence, and abandoning a correct answer because a user or other source pushes back.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesThe benchmark distinguishes four attributions for the initial claim:
- Self: the model stated the claim in an earlier turn.
- User: the claim was attributed to the user.
- Document: the claim came from a supplied document.
- Unattributed: no source is specified.
The point is not to reward agreement with one particular source. A reliable correction should matter more than an unsupported challenge, regardless of who made the original claim.
How the benchmark is structured
Mishra describes SoBA as a multi-turn evaluation built with Kaggle’s benchmarking SDK. A trial establishes an initial claim, may insert neutral conversational filler, then presents an attribution and counter-evidence before asking for a final structured answer. The article says the benchmark uses fictional knowledge to reduce the chance that familiar facts from model training determine the result. Its example domains include distributed systems, deep-space exploration, biotechnology, and fictional geopolitics and history.
The article says the test varies five factors:
| Factor | Conditions described | Why it matters |
|---|---|---|
| Belief attribution | Self, user, document, or unattributed | Tests whether the source attached to the initial claim affects later belief revision. |
| Source reliability | Reliable or unreliable | Distinguishes warranted correction from credulity toward a weak source. |
| Recency | Recent or outdated | Checks whether a model accounts for stale information rather than treating all evidence as equally current. |
| Repetition | One or five presentations | Tests whether hearing a claim repeatedly changes the answer even without stronger evidence. |
| Conversation depth | Immediate or delayed follow-up | Checks whether a correction still has an effect after intervening turns. |
The author also describes adversarial trap cases: unsupported user pushback, outdated official documents, and fresh but unverified rumors. These are intended to expose an “always flip” strategy, where a model changes its answer whenever challenged rather than judging the quality of the evidence.
What the metrics mean
The article names five measures. Final Accuracy records whether the final answer is correct. Persistence Error Rate (PER) concerns persistence in the face of counter-evidence; False Revision Rate (FRR) concerns changing an answer when revision is not warranted. The article also reports a Source Sensitivity Index (SSI), though its brief description does not provide enough detail to reconstruct the index independently.
The headline measure is Self-Authority Bias (SAB), defined by Mishra as the difference between PER under self-attribution and PER under document attribution. A positive SAB means more persistence error when the contradicted claim is attributed to the model itself than when it is attributed to a document. A negative value points in the opposite direction. SAB is a comparison between two conditions, not a general measure of intelligence, honesty, or answer quality.
What the article reports—and what it does not establish
Mishra reports 760 balanced multi-turn trials, with 16% designated as adversarial trap items, and says the reported metrics use 95% bootstrap confidence intervals. The figures below are claims in the September 25, 2026 article; they have not been independently validated here. The profile names are labels in that submission, not evidence about particular commercial systems.
| Profile named in the article | Reported results | Interpretation and qualification |
|---|---|---|
| Calibrated Reasoner | 89.2% final accuracy; 8.9% PER; 11.4% FRR; +68.2% SSI; +6.6% SAB, with reported interval −2.1% to +17.4% | The reported SAB interval spans zero, so these figures do not clearly establish a positive self-attribution effect for this profile. |
| Self-Protective Stubborn Sloth | 90.4% final accuracy; +41.8% SAB, with reported interval +22.4% to +59.2%; PER of 23.4% under document attribution and 65.2% under self-attribution | The article presents a large difference in persistence error between the two attribution conditions despite high reported final accuracy. |
| Sycophantic Agent (FlipFlop) | 45.1% final accuracy; 13.4% PER; 67.6% FRR; +27.6% SSI; −4.0% SAB, with reported interval −16.9% to +11.0% | The high reported FRR is consistent with the profile label’s emphasis on over-revision; the SAB interval spans zero. |
| Repetition-Biased Reasoner | 49.9% final accuracy; 8.4% PER; 63.0% FRR; +23.5% SSI; +0.2% SAB, with reported interval −10.5% to +11.0% | The article reports a near-zero SAB with an interval spanning zero, alongside high FRR. |
The article additionally reports an 18.4% increase in false revision when an unreliable rumor was repeated five times. That is a benchmark-reported result, not a general estimate of how often deployed AI systems are swayed by repeated rumors.
How to read the result without overclaiming
SoBA’s framing is useful because “trust” is not a single behavior. A model that refuses to correct its own prior answer may look consistent while being stubborn; a model that readily accepts user corrections may look responsive while being sycophantic. Testing reliable and unreliable evidence together helps separate careful revision from automatic agreement or refusal.
At the same time, the article’s numbers should be read as an account of a particular benchmark submission. They do not show that AI systems generally trust themselves more than users, nor do the profile labels identify measured behavior of named products. The reported confidence intervals also matter: for the Calibrated Reasoner, Sycophantic Agent, and Repetition-Biased Reasoner SAB results, the stated intervals include zero; the Self-Protective Stubborn Sloth interval does not.
Limitations and what a stronger test would add
Mishra identifies synthetic knowledge as a way to isolate belief revision from familiar facts, but notes that it cannot reproduce the full complexity of real-world credibility judgments. The described first version tests direct contradictions within one session. Cross-session memory and semantically subtle contradictions are presented as future directions, not capabilities already established by the reported evaluation.
For readers assessing any benchmark of belief revision, the key questions are whether it varies source reliability as well as source identity, tests freshness and repetition, checks delayed corrections, and penalizes both stubbornness and unwarranted reversals. Results are more informative when the dataset, scoring definitions, code, and evaluation conditions can be inspected and independently rerun. The article points to a Kaggle notebook, dataset, and code path, but their availability, licensing, execution, and current SDK compatibility are not established by the article’s description.
What the benchmark can tell you
SoBA offers a way to ask a precise question: does the attribution of an earlier claim change how a model responds to later evidence? Its reported examples illustrate why answer accuracy alone is not enough—one system could reach a high final accuracy while still showing a marked attribution-linked persistence pattern. But the available figures support conclusions about the benchmark submission and its described profiles, not a broad verdict about AI systems or their trust in people.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
