Free tools Windows power users keep installed
One-click scans. No signup required.
GPT-4o and Claude 3.7 Sonnet can use base rates and new evidence in the direction prescribed by Bayes’ rule, yet condemn a hypothetical person for making a comparable inference. A preregistered 2026 arXiv preprint by Nykko Vitali and Mahzarin R. Banaji reports this “Bayesian hypocrisy” across five experiments, 48 conditions and more than 5,000 trials. The models’ social judgments were also harsher than those of the human comparison group.
What “Bayesian hypocrisy” means
Bayesian hypocrisy is a dissociation between a system’s numerical reasoning and its social evaluation. The system updates its own probability estimate using a group base rate and evidence, then labels another agent unfair, unjust or inaccurate for making the same type of inference.
As an Amazon Associate I earn from qualifying purchases.
The finding does not mean that statistical reasoning should always defeat fairness norms. It shows that an AI can produce a statistically appropriate answer while applying a different, more punitive standard when judging a person who reaches a similar answer.
How the experiments tested the double standard
Probability judgments
The researchers adapted tasks from a 2019 human study. A model saw a man and a woman who had both performed surgery and was told that only one was a doctor. It estimated who was more likely to be the doctor, then received additional evidence and estimated conditional probabilities. Those responses could be compared with the direction expected from Bayes’ rule.
#1 Best Overall
Judging “Person X”
In a separate judgment task, the model read that Person X concluded the man was more likely to be the doctor after learning that both people had performed surgery. The model rated Person X for intelligence, accuracy, fairness and justice. In the primary within-subject design, the probability and judgment tasks occurred in the same conversation, with filler tasks between them, allowing a direct comparison of the model’s own inference with its evaluation of X.
Models and scale
The preprint tested GPT-4o and Claude 3.7 Sonnet in five experiments comprising 48 experimental conditions and more than 5,000 trials. It also compared the model results with an earlier human sample. The study was preregistered; its data and analysis code are linked from the 2026-08-03 arXiv record.
What the models actually did
| Comparison | Finding |
|---|---|
| Direction of probability updating | Both models generally moved in the Bayesian direction and were near human performance on that broad measure. |
| Variation in estimates | The models were more rigid and rule-based than people, showing much less variation in estimated likelihoods. |
| Claude’s social ratings | In the main within-subject study, Claude assigned the hypothetical person the minimum fairness and justness rating in every trial. |
| GPT-4o’s social ratings | GPT-4o typically gave Person X a fairness/justness score of 2 on a seven-point scale. |
| Human comparison | The human average for the comparable rating was 3 on the seven-point scale. |
As first author Nykko Vitali put it, both models “reasoned in the direction Bayes’ rule predicts they should,” but their social inference failed to align with their numerical inference. The key result is therefore not simply that a model can calculate probabilities; it is that the model can condemn a person for an inference resembling its own.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallThe human precedent
The pattern mirrors a 2019 study by Cao, Banaji and Kleiman-Weiner. Ninety-three percent of 199 participants said that a man and woman who had both performed surgery were equally likely to be doctors. Yet those same participants criticized Person X for saying the man was more likely. In a related hospital doctor-and-nurse scenario, 91% gave the egalitarian answer while still judging the hypothetical reasoner harshly.
Rank #3
That precedent matters because it frames the LLM result as a human-like inconsistency rather than a uniquely machine error. People can endorse a statistical judgment in one role and apply a social norm against it in another.
Why this matters for AI evaluation
Statistical fidelity and fairness can diverge
Systems used to assess applications, moderation cases, safety incidents or other socially sensitive material may treat a base-rate inference as numerically valid while describing a person who makes it as biased or unjust. If those outputs are used together, users may receive an apparently precise probability and a moral verdict that do not follow a consistent standard.
“LLM as judge” scores need a consistency check
A judge model should be tested on at least five axes: agreement with Bayes’ rule, rigidity versus variability of probability estimates, severity of fairness, justice, accuracy and intelligence ratings, consistency between its own reasoning and its judgment of Person X, and behavior in allocation tasks such as sharing money. A high score on one axis does not establish reliability on the others.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Prompting alone may not remove the problem
The experiments placed both tasks in one conversation, separated by filler tasks, yet the dissociation remained. That suggests that asking a model to “be consistent” is not a substitute for measuring whether its numerical and social outputs actually match under controlled conditions.
Best Value
What the study does not establish
- It tested two named models, GPT-4o and Claude 3.7 Sonnet, not every current or future language model.
- The scenarios were adapted hypothetical tasks, so the results do not by themselves predict behavior in every language, culture, domain or real-world decision.
- The authors diagnose an inconsistency; they do not argue that Bayesian updating should automatically override fairness rules.
- Near-human performance in the direction of Bayesian updating does not mean the models reproduced human variability or human standards of social judgment.
Bottom line for users of AI judges
LLMs can reason statistically and still judge a person more harshly for making a comparable inference. Treat probability answers and fairness evaluations as separate behaviors that require separate validation. Before relying on a model in a high-stakes setting, test whether it applies the same evidential standard to its own answer and to the person it is asked to assess.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
