Free tools Windows power users keep installed
One-click scans. No signup required.
OpenAI’s 2024 SimpleQA benchmark found that several tested models answered short factual questions correctly less than half the time, but that is not a universal error rate for AI. The results apply to named model versions on a narrow, no-browsing test—and they distinguish wrong answers from questions a model declined to answer.
What SimpleQA tested
OpenAI introduced SimpleQA on October 30, 2024, as an open benchmark for language-model factuality. Its 4,326 questions ask for short, verifiable answers across subjects including science and technology, television, and video games. The benchmark was designed to challenge frontier models while keeping answers relatively straightforward to evaluate. OpenAI’s benchmark description explains its design and results.
| # | Preview | Product | Price | |
|---|---|---|---|---|
| 1 |
|
AI Explained: Facts, Fiction, and Future | $12.99 | Buy on Amazon |
| 2 |
|
History Facts: Inventions From Fire To Ai (Facts and Knowledge) | $15.99 | Buy on Amazon |
| 3 |
|
Simply AI (DK Simply) | $14.20 | Buy on Amazon |
| 4 |
|
Just the Facts - Theory Workbook - Book 1 | $12.95 | Buy on Amazon |
As an Amazon Associate I earn from qualifying purchases.
To build the set, trainers researched questions and answers, then a second trainer answered each question independently. OpenAI included questions only when the answers matched. A third trainer reviewed a random sample of 1,000 questions; after examining disagreements, OpenAI estimated that about 3% of the dataset had an inherent error. That is the benchmark authors’ estimate of possible flaws in the questions or reference answers—not a measure of model mistakes.
How to read the scores
SimpleQA grades responses as “correct,” “incorrect,” or “not attempted.” A response that neither supplies the reference answer nor contradicts it is classed as not attempted. A response that contradicts the reference answer is incorrect, even if it includes hedging. That distinction matters: a model can avoid a wrong answer by abstaining, but abstention is not the same as answering correctly.
#1 Best Overall
OpenAI’s later o1 system card, published December 5, 2024, reports accuracy and hallucination rates for specific versions tested on SimpleQA:
| Model version | Accuracy | Hallucination rate |
|---|---|---|
| GPT-4o | 0.38 | 0.61 |
| o1 | 0.47 | 0.44 |
| o1-preview | 0.42 | 0.44 |
| GPT-4o-mini | 0.09 | 0.90 |
| o1-mini | 0.07 | 0.60 |
These are the system card’s reported figures for its evaluation, not scores for every version in those model families or for current models generally. Accuracy and hallucination rate are separate measures; neither alone describes all aspects of reliability. OpenAI’s original 2024 benchmark publication also says smaller tested models answered fewer questions correctly than GPT-4o and o1-preview, while the o-series models more often returned “not attempted.”
What the “42.7%” headline means
A November 2, 2024, Futurism article reported o1-preview’s SimpleQA success rate as 42.7%. OpenAI’s later system-card table gives its accuracy as 0.42. Both figures concern o1-preview on this benchmark; neither should be read as a timeless estimate of how often AI gives wrong answers. The result also does not mean that the remaining responses were all wrong: the grading scheme distinguishes incorrect answers from not-attempted responses.
Can models tell when they don’t know?
SimpleQA’s results show why correctness and willingness to abstain need separate consideration. OpenAI reported that the o-series models more often chose “not attempted,” but that does not by itself show how well a model recognizes every situation in which it lacks an answer.
Rank #3
OpenAI also examined confidence. In its analysis, stated confidence was positively related to accuracy, so confidence contained some signal. But the models were not perfectly calibrated and tended to overstate confidence on average. That does not mean every confident answer is false; it means confidence should not be treated as a guarantee of correctness.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What these findings do—and don’t—establish
SimpleQA tests short questions with one intended, verifiable answer. It does not directly measure performance on long-form writing, where a response may contain many factual claims, or establish reliability in specialist work, changing-current-information questions, or web-enabled answers. OpenAI explicitly said that whether success on short factual answers correlates with the ability to write lengthy responses containing numerous facts remains an open research question.
Quick Recap
- The scores are evidence about the specific model versions and evaluation setup reported in 2024, not a ranking of models available today.
- The benchmark’s roughly 3% estimated dataset error is OpenAI’s assessment of the question set, not the models’ error rate.
- Accuracy, hallucination rate, abstention frequency, and confidence calibration describe different behaviors. A single number cannot capture them all.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
