AI reliability is improving on some measured tasks, but the evidence does not show a clear industry-wide trend or prove that progress has plateaued. Results depend on what counts as a hallucination, which questions are asked, whether a model can abstain, and how its answers are scored. Some developer-reported comparisons show fewer factual errors; other studies show that hard prompts still expose weaknesses and that older benchmarks can stop distinguishing newer systems.
What does “improving reliability” actually mean?
“Hallucination” is often used as a catch-all for invented details, unsupported claims, or answers that sound certain but are wrong. Those failures are not measured in one standard way. An evaluation might count errors in individual factual claims, mark a whole response wrong if it contains a major error, or measure whether an answer matches a known correct response. Those metrics have different denominators and cannot be treated as interchangeable.
As an Amazon Associate I earn from qualifying purchases.
Nor are all questions alike. A short factual question, a long biography, a summary grounded in a supplied document, and an obscure or adversarial prompt test different capabilities. A system that performs well on one kind of test may still fail another. Reliability also includes calibration: whether the model signals uncertainty or declines to answer when it lacks a sound basis.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
What the current evidence does—and does not—show
| Evidence | Reported result | What it supports | What it does not establish |
|---|---|---|---|
| OpenAI’s GPT-5 system card (2025) | OpenAI reports a 26% lower factual-claim hallucination rate for GPT-5-main than GPT-4o, and a 65% lower rate for GPT-5-thinking than OpenAI o3, in the card’s specified evaluations. | Some tested models performed better than selected predecessors on those factuality measures. | An independently verified improvement across providers, tasks, or real-world use. |
| Stanford HAI’s 2026 AI Index | A new accuracy benchmark reported hallucination rates from 22% to 94% across 26 models. | Measured reliability can vary greatly across models on a particular benchmark. | A universal hallucination rate for everyday interactions or every kind of task. |
| FactBench (ACL 2025) | The authors evaluated 1,000 prompts across 150 topics, selected to elicit factual errors and tiered by difficulty. They found factual precision declined from easy to hard prompts. On this evaluation, a larger Llama 3.1 model performed comparably to or worse than its 70B variant. | Prompt difficulty matters, and greater model scale did not guarantee better factual precision in this tested comparison. | A general rule that larger models are less factual, or that scale never helps. |
| Akhtar et al., PMLR 306 (2026) | In an analysis of 60 language-model benchmarks, nearly half exhibited saturation; saturation rates increased with benchmark age. | Some benchmarks lose their ability to distinguish model performance as they age. | That practical AI reliability itself has stopped advancing. |
| Nature study (2026) | The authors argue that raw accuracy or pass-rate scoring can reward a model for guessing rather than abstaining when uncertain. | Scoring rules influence whether an evaluation rewards trustworthy uncertainty or simply more answers. | That a higher answer rate alone means a system is more reliable. |
| NIST AI 800-3 (2026) | The report examines three benchmarks and 22 API-access frontier language models, and discusses statistical methods for estimating generalized accuracy and uncertainty. | Benchmark scores and their uncertainty need careful statistical treatment. | That this set of models represents the entire market or all deployment settings. |
The GPT-5 comparisons are developer-reported results from OpenAI’s own system card, not an independent, harmonized comparison of the industry. Stanford’s wide benchmark range, meanwhile, describes outcomes on its new accuracy benchmark—not a single rate users should expect in all conversations.
#1 Best Overall
Why the answer changes with the test
Prompt difficulty and task type
FactBench’s results illustrate why an average score can conceal weak spots: factual precision fell as its prompts became harder. A system evaluated on straightforward recall may look stronger than the same system facing obscure, open-ended, or error-inducing questions. Document-grounded summarization adds another condition: the evaluator must distinguish a claim supported by the supplied material from one that merely sounds plausible.
Whether the model is allowed to say “I don’t know”
A model that answers every question can accumulate more correct answers while also making more unsupported guesses. The 2026 Nature analysis highlights this scoring problem: raw accuracy or pass rate can favor guessing over abstention. A useful evaluation should therefore make clear how it treats unanswered questions and should account for both correctness and the decision to abstain. Coverage—the share of questions the model attempts—matters, but it is not a substitute for trustworthiness.
How graders define and verify errors
Claim-level error rates and the share of responses containing at least one major error answer different questions. Results can also depend on whether claims are checked against sources, who or what grades them, and whether automated grading is validated. A benchmark score without its error definition and grading method is too thin a basis for comparing reliability.
Does benchmark saturation mean AI reliability has plateaued?
No. Saturation means a benchmark may have become less useful for separating systems—for example, because newer models converge on its tasks or its design has aged. It is a warning about the measuring instrument, not proof that models have stopped getting more reliable on useful tasks. The PMLR 2026 analysis found saturation in nearly half of the 60 benchmarks it studied, with rates rising with benchmark age; it did not establish a flat real-world reliability trend.
There are three claims to keep separate:
- Some measured factuality results have improved: supported by OpenAI’s reported comparisons among selected models on specified evaluations.
- Some benchmarks have saturated: supported by the PMLR study’s analysis of benchmark behavior.
- Everyday reliability across providers has a clear upward or flat trajectory: not resolved by this evidence. The available results do not form one independent, harmonized time series using the same tasks and error definitions over multiple years.
How to judge a claim that a model is “more reliable”
Before taking a headline score or model comparison at face value, check what it actually measures:
- Task and difficulty: Is the test short factual recall, open-ended generation, document-grounded work, or hard prompts designed to expose errors?
- Error definition: Is the reported number about claims, complete answers, major errors, or correct responses?
- Abstention and coverage: Can the model decline to answer, and how is that scored alongside incorrect answers?
- Grading evidence: Are answers checked against sources? Who grades them, and is the grading method validated?
- Benchmark freshness: Does the test still distinguish systems, or has it become saturated? The PMLR analysis associates expert curation with resilience among the benchmarks studied, but this does not make any one test a universal measure.
- Uncertainty: Does the report explain how much confidence to place in the score? NIST’s 2026 report warns that some common metric approaches can yield invalid uncertainty estimates or rely on unrecognized assumptions.
- Comparison scope: Is this an independent comparison across providers, or a developer’s report comparing its own systems under its own stated evaluation?
These checks do not produce one definitive reliability number. They help establish whether a comparison is relevant to the task a reader cares about and whether its conclusions extend beyond the benchmark.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.So, are hallucinations improving or are we plateauing?
The most defensible answer is uneven progress, not a demonstrated industry-wide plateau. Selected evaluations show reductions in factual errors, while difficult prompts, scoring choices, and benchmark limitations complicate broader claims. To establish a genuine long-term trend, comparisons would need to track the same kinds of tasks and error definitions across providers and years, account for abstention and grading uncertainty, and use tests that continue to distinguish systems.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteQuick Recap
Best Value
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
