ARC-AGI tests whether an AI can infer a rule from a few examples of unfamiliar visual puzzles and apply it to a new case. MMLU, GPQA, Humanity’s Last Exam (HLE), and SWE-bench ask different questions about academic knowledge, scientific problem-solving, expert-level exams, and software engineering. Their scores are evidence about different abilities—not points on one shared scale of “reasoning.”
What does ARC-AGI measure?
ARC is the Abstraction and Reasoning Corpus, associated with François Chollet’s proposal to evaluate how well systems generalize to novel tasks. In its original format, a task presents small colored grids with example inputs and outputs. The solver must infer the transformation behind those examples, then apply it to a new input. The central challenge is rule induction from limited examples, rather than answering from a large store of subject facts.
As an Amazon Associate I earn from qualifying purchases.
The ARC-AGI-1 repository describes the benchmark through several lenses: general intelligence, program synthesis, and psychometric intelligence testing. Those framings help explain what its tasks are meant to probe, but a benchmark provides a narrow experimental window into capability; it is not a complete measure of intelligence. See the ARC-AGI-1 repository.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
How ARC-AGI-1 and ARC-AGI-2 differ
ARC-AGI-2 is a distinct edition, designed to probe greater cognitive complexity through combinations or interactions among rules. Its design discussion highlights symbolic interpretation, compositional reasoning, and contextual rules. The technical report also describes first-party human testing as a way to compare human and AI performance. See the ARC-AGI-2 materials and the technical report.
#1 Best Overall
| Edition | Trials per test input | What to keep in mind |
|---|---|---|
| ARC-AGI-1 | Three, according to its repository | Use the edition and evaluation conditions when interpreting a score. ARC-AGI-1 repository |
| ARC-AGI-2 | Two, according to its repository | Its materials and protocol differ from ARC-AGI-1. ARC-AGI-2 repository |
ARC-AGI-1 and ARC-AGI-2 have different evaluation sets, protocols, and human baselines. A score from one edition does not transfer directly to the other, so do not treat them as interchangeable or combine them into a trend without accounting for the conditions. The ARC-AGI reference page cautions against direct score transfer across editions.
What each benchmark asks
“Reasoning” covers many task types. These benchmarks differ in what goes in, what a system must produce, and how much the task relies on domain knowledge. The comparisons below are qualitative; check each evaluation’s methodology before comparing particular scores.
Rank #2
| Benchmark | Task family | How it contrasts with ARC-AGI |
|---|---|---|
| ARC-AGI | Inferring and applying rules to novel visual grid tasks | Emphasizes generalization on compact, unfamiliar transformations rather than broad factual recall. |
| MMLU | Academic subject knowledge and multiple-choice question answering | Relies more on language-based exam performance and stored knowledge than ARC’s visual rule-induction tasks. |
| GPQA | Graduate-level science questions designed to resist ordinary web lookup | Tests demanding scientific knowledge in question-and-answer form, not visual transformations. |
| Humanity’s Last Exam (HLE) | Structured academic problems contributed by subject experts | It is an academic examination benchmark, not a direct test of flexible visual rule induction. Its official page says it is not a test of open-ended research or creative problem solving: HLE official page. |
| SWE-bench | Software engineering task performance | Measures coding and software work, a different task family from ARC’s abstract visual puzzles. |
The benchmark descriptions for MMLU, GPQA, and SWE-bench here are intentionally high-level; the available comparison references do not establish a common, standardized protocol across all the listed benchmarks. For a broader benchmark overview, see the Stanford AI Index.
Recommended Free Tools
How to compare benchmark results responsibly
A percentage alone leaves out important context. Before interpreting a score or comparing two systems, check the evaluation conditions that produced it:
- Edition and test set: Identify the benchmark version and split. ARC editions are not interchangeable.
- Input and output: Distinguish visual transformations from multiple-choice exams, science questions, or code changes.
- Attempts and tools: Record how many tries were allowed and what tools or external resources a system could use. ARC repositories specify different trial limits by edition.
- Model and compute: Check the model configuration, sampling method, and compute budget used for the result.
- Human comparison: Confirm which human baseline and evaluation set are being used, especially when comparing ARC editions.
These factors can change what a score means. There is no single standardized protocol shared across all the benchmarks in this comparison, so a leaderboard ranking should not be read as a direct measure of general reasoning ability.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What a high ARC-AGI score can—and cannot—show
A strong result is evidence that a system performed well on the specified ARC edition under its stated conditions. It can indicate skill at the benchmark’s visual rule-induction and generalization tasks. It does not, by itself, establish broad competence across academic knowledge, scientific problem-solving, software engineering, or every other meaning of reasoning. Nor can one benchmark settle whether a system is generally intelligent.
For a fuller picture, read ARC alongside benchmarks that test other task families, and compare results only when their versions and evaluation conditions are clear.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsQuick Recap
Best Value
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
