There is no single score that captures how persuasive a language model is. Researchers measure different outcomes: whether people rate one text as more persuasive than another, whether an argument shifts readers’ stated attitudes, or whether a model can move a conversation partner’s position in an interactive or simulated contest. Those outcomes are related, but they are not interchangeable.
To interpret a result, first ask what changed, for whom, after what kind of exposure, and against which baseline. A benchmark score about persuasive wording does not prove that readers changed their minds; a short-term change in a survey answer does not establish a lasting change in behavior.
As an Amazon Associate I earn from qualifying purchases.
What does “persuasive” mean in a study?
Persuasiveness can refer to a quality readers perceive in a message or to an effect the message has on a person. The distinction matters: a fluent, compelling-sounding argument may not change anyone’s view, while a measured shift in agreement does not by itself show that the argument’s wording was unusually persuasive.
- Persuasive language: People judge which of two texts uses more persuasive language. This evaluates the text, not necessarily its effect on an audience.
- Attitude change: Researchers compare a participant’s stated position before and after exposure to an argument.
- Interactive persuasion: A person or model tries to change a partner’s position through a conversation or debate.
- Simulated persuasion: A model faces a task or opponent under benchmark rules. Success is evidence about that setup, not a direct estimate of real-world influence.
A study’s outcome should therefore be named precisely. “Scored higher on a text-persuasion benchmark” and “shifted participants’ stated support” describe different findings.
#1 Best Overall
How do researchers measure changes in people’s views?
Pre- and post-message attitude surveys
In a common design, participants report how much they agree with a claim, read an argument supporting it, and report their agreement again. The difference between the two answers is the measured shift. This captures short-term stated attitude change under the study’s conditions; it does not establish that the person will act differently or retain the view later.
Anthropic’s April 2024 study used 28 topics, with a supporting and opposing claim for each, for 56 claims in total. It recruited 3,832 unique participants. Three people evaluated each claim-argument pair, and researchers averaged their viewpoint shifts into an aggregate persuasiveness measure. The topics were primarily complex and emerging issues, selected in part because researchers expected views to be less hardened. That choice helps define what the results can speak to: performance on those arguments and topics is not automatically evidence about entrenched beliefs or every subject area.
Human-written arguments as a baseline
A model score is easier to interpret when it is compared with a meaningful alternative. Anthropic compared model-generated arguments with arguments written by human participants. In that study’s evaluation, Claude 3 Opus was not statistically significantly different from the human writers on the persuasiveness score. This is a result about that model, those arguments, and that study design—not proof of universal parity between models and people.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Single-message effects depend on the sample and measure
A 2025 Nature Human Behaviour study of single-message political persuasion recruited 25,982 US adults online in April and May 2024. Its abstract reports an estimated average attitude shift of 5.77 percentage points for a model of average size in that sample. The figure belongs to that experiment’s political issues, audience, and measurement; it should not be generalized to every model, population, or decision.
A 2025 Nature Communications report describes three preregistered experiments involving 4,829 participants and effects of about 2–4 points on 101-point policy-support scales. These are modest shifts on the reported scale, not a percentage of people persuaded. The study’s abstracted result does not, on its own, establish whether changes persisted or translated into behavior.
How do text benchmarks and contests differ?
Paired-text judgments measure relative wording
The 2025 NAACL paper by Amalie Brogaard Pauli, Isabelle Augenstein, and Ira Assent introduced PERSUASIVE-PAIRS, a dataset of 2,697 short-text pairs annotated for relative persuasive language. The authors trained a regression model to score the relative difference between paired texts. They report that different LLaMA 3 system-prompt personas changed persuasive language even when the task instruction was only to paraphrase.
This benchmark tests how persuasive the language in paired texts is judged to be. It does not directly test whether readers’ attitudes changed. The finding about paraphrasing also illustrates why prompt and persona conditions need to be reported: apparently neutral transformations can affect the property a benchmark scores.
Simulated and generated persuasion are separate tasks
The 2025 ICLR paper by Somesh Singh, Yaman Singla, Harini S I, and Balaji Krishnamurthy introduced PersuasionBench and PersuasionArena, distinguishing simulated persuasion from generative persuasion. Its abstract reports simulated capability barely above random, while generated persuasion performs better. In one reported comparison, GPT-4o lost only 36% of the time against the best human persuader. The paper also reports a positive correlation between model size and persuasiveness, while finding that targeted training could make smaller models more persuasive than much larger ones.
These are benchmark-specific results. Rules, opponent selection, prompts, and scoring determine what a contest measures; a win rate in that setting is not a population-level estimate of persuasion in everyday conversations.
Rank #4
What changes when persuasion is interactive?
A fixed argument gives every participant the same exposure. A conversation can adapt to what a person says, and personalization can change the message further. Those differences affect what a study is testing.
A 2025 Nature Human Behaviour interactive-persuasion study reports an experiment with 900 participants across human-human and human-AI conversation conditions, including personalized conditions. Persuasion was measured by agreement before and after debates. The interaction format and personalization are therefore part of the result, not incidental details to ignore when comparing it with a single-message experiment.
Recommended Free Tools
Large-scale evidence also points to the importance of model conditions. The UK AI Security Institute’s 2025 study involved 76,977 participants, 707 political issues, and 19 language models. It reports that post-training increased persuasion by up to 51% and prompting by up to 27% in the tested conditions. The institute also reports that where these methods increased persuasion, they systematically decreased factual accuracy. “Up to” describes the maximum reported effect, not a typical improvement for every model or issue. The accuracy trade-off means a persuasion result should be considered alongside what the model said and whether it was true.
Best Value
How should you compare two persuasion results?
Before treating two scores as a ranking, check whether the studies measured the same thing under comparable conditions. A useful comparison records:
- Outcome: Was the measure a relative text rating, stated attitude shift, behavioral choice, or success in a simulated contest?
- Audience and issue: Who participated, where were they recruited, and were the issues familiar or potentially entrenched?
- Exposure: Did participants read one fixed message, take part in a conversation, or receive personalized arguments? How much time did the interaction allow?
- Model conditions: Which model and version were used, and what system prompt, prompting strategy, or post-training was applied?
- Baseline: Was the comparison against human-written arguments, experts, a neutral control, or another model?
- Validity checks: Were uncertainty, factual accuracy, persistence, and behavioral outcomes assessed?
If these features differ, the results may answer different questions rather than show that one model is categorically more persuasive. A short-term survey shift, a text-rating score, and a contest outcome should not be collapsed into one universal leaderboard.
What can the evidence establish—and what can’t it?
Experiments can show that, under particular conditions, a message or conversation changed participants’ stated views, or that evaluators preferred one text’s persuasive language. Benchmarks can compare systems consistently within their defined tasks. Together, these methods help identify how topic, prompting, training, and interaction format affect measured persuasion.
They do not automatically show that effects last, change decisions, or alter real-world behavior. Anthropic’s 2024 study explicitly cautions that laboratory findings may not transfer to real-world settings and identifies effects on decisions and actions as further questions. The most defensible conclusion is therefore bounded: name the model, task, audience, exposure, baseline, and measured outcome whenever reporting a persuasion result.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
