Yes. In three 2026 studies, an AI model’s verdict on the same moral case moved when only the presentation changed: the wording, the order of answer options, the labels attached to those options, the narrator’s point of view, or the protocol used to score the answer. The facts and the moral conflict were meant to stay constant. The sensitivity is measurable, but it does not come from a single mechanism, and a changed answer does not automatically mean the model changed its underlying stance.
What counts as moving the boundary
The studies separate several kinds of change that are easy to lump together. Each one is a place where the boundary of a verdict can shift without the case itself shifting. Only some of these axes alter the story; others change how the model is asked or how its answer is scored.
As an Amazon Associate I earn from qualifying purchases.
| Change made | What is meant to stay fixed | Reported result | Source |
|---|---|---|---|
| Surface rewording of the same dilemma | Facts and moral conflict | 7.5% flip rate on generated perturbations, within the authors’ stated self-consistency noise floor of 4–13% | Van Nuenen and Sachdeva, arXiv preprint, 2026 |
| Point-of-view shift | Underlying events; the narrator changes | 24.3% instability | Van Nuenen and Sachdeva, arXiv preprint, 2026 |
| Answer order and option labels in a yes/no question | Case text | Order and lexical components reported per model (see the model table below) | Haonan Huang, arXiv paper, 2026 |
| Response format: graded rating against binary verdict | Case text | Cross-form incoherence of 0.12–0.21 on a ±1 axis for graded ratings by the tested frontier models | Haonan Huang, arXiv paper, 2026 |
| Evaluation protocol for an automated judge | Model and scenario | 67.6% agreement between structured protocols (κ=0.55); 35.7% of model-scenario units matched across all three protocols | Van Nuenen and Sachdeva, arXiv preprint, 2026 |
Does an AI give the same verdict if I reword the question?
The largest sample among the three studies is van Nuenen and Sachdeva’s preprint. They evaluated 2,939 dilemmas drawn from the r/AmItheAsshole forum, posted between January and March 2025, using four models and 129,156 judgments. Surface perturbations, meaning rewordings that kept the same story, flipped verdicts 7.5% of the time. That figure sits inside the 4–13% range the authors give as their self-consistency noise floor, so in this setup small rewordings produced changes that were not clearly larger than the models’ own inconsistency.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesThat is narrower than saying wording does not matter. It applies to the perturbations the authors generated and to the four models they tested. Rewordings that change the emphasis of a story, rather than its surface, are a different test.
#1 Best Overall
Can changing the point of view change an AI’s moral judgment?
It can, and by more. When the same events were retold from a different party’s perspective, van Nuenen and Sachdeva recorded 24.3% instability, more than three times the surface-perturbation figure. The authors draw a broader conclusion from this pattern. In their words: “These results show that LLM moral judgments are co-produced by narrative form and task scaffolding, raising reproducibility and equity concerns when outcomes depend on presentation skill rather than moral substance.” This is the authors’ own statement in a preprint, not an official consensus position, and it rests on the setup described above.
Is the judge reacting to the facts or to how I asked?
Haonan Huang’s 2026 arXiv paper separates the parts of a yes/no answer that come from the question’s logic from the parts that come from its form. Two effects are named. Order bias is a tendency tied to which option appears first. Lexical pull is a tendency tied to the words used for the options. For graded ratings, the paper reports cross-form incoherence of 0.12–0.21 on a ±1 axis for the tested frontier models. That is a measure defined in this study, not a general reliability score.
Rank #2
- 𝐑𝐄𝐒𝐄𝐓 𝐘𝐎𝐔𝐑 𝐌𝐈𝐍𝐃 𝐈𝐍 𝟔𝟎 𝐒𝐄𝐂𝐎𝐍𝐃𝐒 – A simple, screen-free way to disconnect after a high-demand workday or regain focus during a busy afternoon. Pull one of these mindfulness cards, pause, and follow a practical prompt designed to bring calm, clarity, and grounding in about a minute—no app, journal, or meditation experience needed.
- 𝐅𝐈𝐍𝐃 𝐓𝐇𝐄 𝐂𝐀𝐋𝐌 𝐘𝐎𝐔 𝐍𝐄𝐄𝐃 𝐓𝐎𝐃𝐀𝐘 – Includes 52 color-coded prompts across Focus, Calm, Gratitude, Self-Compassion, and Presence. These mindfulness cards for adults make it easy to choose the category that fits the moment, or pull a card at random for a quick daily ritual inspired by approachable mindfulness and grounding practices.
- 𝐁𝐔𝐈𝐋𝐃 𝐀 𝐒𝐄𝐀𝐌𝐋𝐄𝐒𝐒 𝐂𝐀𝐋𝐌𝐈𝐍𝐆 𝐇𝐀𝐁𝐈𝐓 – Keep these self care cards on your desk to break the midday work loop, in your bag for travel, or on your nightstand to transition peacefully into sleep. These bite-sized practices fit naturally into work breaks, quiet mornings, evening wind-downs, and everyday wellness routines.
- 𝐌𝐀𝐃𝐄 𝐓𝐎 𝐅𝐄𝐄𝐋 𝐏𝐑𝐄𝐌𝐈𝐔𝐌, 𝐔𝐒𝐄𝐃 𝐃𝐀𝐈𝐋𝐘 – Crafted from thick 350 GSM cardstock with a smooth premium finish, these cards feel substantial in hand and are designed to withstand repeated shuffling, daily handling, and carrying in a bag or desk drawer without easily bending or creasing. Compact 2.5" x 3.5" size makes them easy to keep close wherever life takes you.
- 𝐆𝐈𝐕𝐄 𝐀 𝐆𝐈𝐅𝐓 𝐓𝐇𝐄𝐘'𝐋𝐋 𝐀𝐂𝐓𝐔𝐀𝐋𝐋𝐘 𝐔𝐒𝐄 – Beautifully designed and easy to use, Mindful Reset makes a meaningful gift for mindfulness, meditation, and daily affirmations. Whether used as meditation cards, affirmation cards, or a simple wellness ritual, this thoughtful deck is perfect for women and men, friends, coworkers, teachers, therapists, students, and loved ones looking to bring more calm and intention into everyday life.
The binary results differ by model. Figures are as reported for the tested models and setup:
| Model (as tested) | Story-averaged binary bias | Order bias component | Lexical pull component |
|---|---|---|---|
| Claude Sonnet | −0.32 | −0.18 | −0.14 |
| Claude Haiku | −0.86 | −0.33 | −0.53 |
| GPT-5.5 | Approximately zero | Not stated in the summary | Not stated in the summary |
| Gemini models tested | Approximately zero | Not stated in the summary | Not stated in the summary |
These numbers describe the apparent yes/no bias for these models and this setup. They are not a ranking of overall judgment quality. A reader comparing the Claude rows should note that the components sum to the reported total, which is a property of the decomposition, not an independent check.
Rank #3
- GO BEYOND SMALL TALK — 52 cards with 104 open-ended questions (two per card) that turn dinners, road trips, and quiet nights in into conversations you'll actually remember. The original Holstee reflection deck.
- TOGETHER OR ON YOUR OWN — spark deeper conversations with couples, families, friends, and coworkers, or use the deck solo as journaling and self-reflection prompts. No rules, no setup — just draw a card and go deeper.
- COLOR-CODED BY THEME — questions span Gratitude, Wellness, Intention, and more, so you can steer toward what matters most in the moment. Inspired by mindfulness and positive psychology.
- SMALL ENOUGH TO POCKET, BEAUTIFUL ENOUGH TO DISPLAY — each card carries a unique, abstract design. Take the deck on the go, or leave it out on the coffee table.
- QUALITY YOU CAN FEEL — made in the USA from sustainably-forested paper with vegetable-based inks and a starch-based laminate that keeps them durable. As kind to the planet as they are to your conversations.
The paper’s most useful distinction is between a flip in the printed answer and a change in the model’s stance. With arbitrary A/B labels, the study found verdict-attached logical bias of approximately zero for every tested frontier model, while surface label and order effects could remain. Huang puts the result this way: “the models are not drawn toward rejecting – the pull follows the printed surface, not the verdict it carries.” In practice, a verdict that flips when you swap the order of the options or the words used for them has moved because of the form, even if the logic of the case is unchanged.
Do automated AI judges agree with each other?
The JudgeSense benchmark, described in a 2026 abstract on alphaXiv, tests judges used to score other outputs. It covers 880 items, four evaluation tasks, and 25 judges from six providers. The abstract reports that rewording reduced agreement on all four tasks, and that the effect reached the authors’ practical-meaning threshold on two of them. The result is what the abstract describes; it does not establish that every judge system behaves in the same way.
Van Nuenen and Sachdeva report a related pattern for protocol changes. Structured evaluation protocols agreed 67.6% of the time (κ=0.55), and only 35.7% of model-scenario units matched across all three protocols they tested. Agreement between protocols is therefore not something to assume from the labels on the output.
How do you test whether an LLM judge is prompt-sensitive?
Huang’s paper frames the goal in one sentence: “Measuring what an AI values requires crossing the frames of the question, not asking once.” The following procedure follows the designs and caveats reported in the studies above. It does not eliminate bias, but it lets you tell whether a verdict moves more than the model’s own noise.
- Write a fact sheet. List the events, actors, and facts of the case as neutral statements, with no judgment words.
- Build equivalent versions. Create at least one reworded version that keeps every fact, and one version told from another party’s point of view. Label them separately, because they test different axes.
- Counterbalance order and labels. For each binary question, run both option orders, and run both label pairs (for example, A/B and then the reverse assignment of your own labels).
- Ask for a graded rating as well as a verdict. State the scale in the prompt, such as −1 to +1, and check whether the rating and the verdict agree.
- Repeat each condition. Run the same prompt several times to estimate the model’s own noise. Compare any flip rate to that baseline, not to zero.
- Log the setup. Record the model name and version, the date, any settings you can control, the system and user prompt text, where instructions are placed, and the response format.
Read the results with a simple rule. A flip on an equivalent rewording that exceeds your repeat-run noise is evidence of prompt sensitivity for that task. A binary flip that follows the order or the label, rather than the content, points to a form effect. A graded rating that disagrees with the verdict points to a format problem, and a judge that fails this check should not be trusted with that task until the cause is found.
What these results do not establish
- They describe the models, cases, and perturbations each study tested, so they do not establish how every model, person, or judge behaves.
- A changed verdict is not, on its own, proof of a changed stance. The order and label analysis is what separates the two.
- The generated perturbations in these studies may differ from the rewrites a real user makes, so their figures should not be read as typical user experience.
- The title’s “boundary” could refer to a legal, philosophical, or technical line. The sources here address judgment sensitivity in AI systems.
The human and legal context is limited. A 2018 analysis of expert witness testimony argues that scientific evidence should be read within the wider context of legal adjudication, and that fact-finders must connect evidence to legal concepts. It also notes that a scientifically validated general proposition does not guarantee the factual and normative correctness of a particular verdict. The AI experiments above are not direct evidence about juries or courts.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
