To evaluate an AI model on ARC-AGI, name the benchmark edition and evaluation split, follow that edition’s scoring protocol, record the model and reasoning configuration, and report accuracy alongside cost and evaluation time. A score without those conditions is difficult to interpret—and scores from ARC-AGI-1, ARC-AGI-2, and interactive ARC-AGI-3 are not directly interchangeable.
What does an ARC-AGI evaluation measure?
ARC-AGI evaluates whether a system can infer a rule from a small number of examples and apply it to a new problem. In ARC-AGI-1 and ARC-AGI-2, tasks use colored grids and input-output transformations. ARC-AGI-2 is intended to test more demanding reasoning, including symbolic interpretation, combining interacting rules, and applying rules differently depending on context. ARC-AGI-3 is interactive, so its results measure performance in a different kind of benchmark environment.
As an Amazon Associate I earn from qualifying purchases.
The ARC Prize Foundation frames the question as not only whether AI can acquire a skill to solve a task, but also at what efficiency or cost. That makes resource use part of a useful evaluation, rather than an optional footnote to accuracy.
Choose the edition and evaluation split
Start by identifying the benchmark generation and the data split. ARC Prize introduced ARC-AGI-2 in 2025, building on the original ARC-AGI-1 format; ARC-AGI-3 changes the format to interactive tasks. A result should not be presented as a single generic “ARC-AGI score.”
| Evaluation set | What it represents | How to report it |
|---|---|---|
| Public | Tasks available for research and development. The ARC-AGI-2 repository lists 1,000 public training tasks and 120 public evaluation tasks. | Name the public evaluation set and edition. Do not imply that a public-set result is a private-set result. |
| Semi-private | A 120-task ARC-AGI-2 test set described for remotely hosted commercial models. | Identify it as semi-private and specify the evaluation protocol used. |
| Private | A 120-task ARC-AGI-2 test set used in the competition. | Identify it as private and distinguish it from public-task experiments. |
The repository reports 66% average human performance on its public evaluation tasks in its test sample. That is a result for that sample, not a promise that every person will achieve that score. ARC Prize says the public, semi-private, and private evaluation tasks were calibrated so each was solved by at least two humans within two attempts. Its early-2025 San Diego calibration study involved more than 400 members of the general public; this describes task calibration, not perfect performance by every participant.
Use the scoring rule for the edition you are testing
For the 2026 ARC-AGI-2 competition, a solver submits exactly two predicted outputs for each test input. If either output exactly matches the expected output, that test output scores 1; otherwise it scores 0. The final score is the average over task test outputs. This is the competition’s pass@2-style exact-match rule, not a universal scoring rule for every ARC-AGI edition or evaluation.
Record the permitted number of attempts and outputs with the result. Changing the attempt budget or scoring rule changes what the score means, even if the model and dataset remain the same.
Run a reproducible evaluation
- Select the edition and split. Use the ARC Prize benchmarking repository and guide for ARC-AGI-1 or ARC-AGI-2, then record the exact dataset split and its exposure status.
- Freeze the system configuration. Record the model name, reasoning level, and token limits. Preserve the code, prompts or task interface, attempt count, and tools allowed by the protocol so another evaluator can understand what produced the result.
- Apply the correct protocol. Follow the relevant edition’s instructions, including output format, number of predictions, and scoring rule. For ARC-AGI-2’s 2026 competition rule, submit exactly two outputs per test input and score exact matches as described above.
- Capture performance and resources. Save the score, individual task results where available, evaluation duration, and cost. State what the cost covers when that is known; cost figures are not comparable if their accounting boundaries or system setups differ.
- Label verification status. Say whether a result is listed as verified by ARC Prize, self-run, or community reported. ARC Prize says submissions are not verified by default and that it selectively verifies results.
- Date and archive the report. Keep the configuration and evaluation conditions with the result, and date the score. ARC Prize says public outputs, evaluation durations, costs, and individual task scores are published for results.
Interpret scores in context
A score is evidence about a particular system under particular conditions—not a standalone measure of general intelligence. Compare two options only when their edition, split, scoring rule, output or attempt budget, and evaluation setup align. Then consider cost per task and total duration as well as accuracy: a higher score obtained with substantially more computation may not be the more efficient system.
For historical context, ARC Prize’s 2026 technical report says the top score in the 2025 global competition was 24% on the ARC-AGI-2 private evaluation set at $0.20 per task. The competition ran from March 26 to November 3, 2025, with 1,455 teams and 15,154 entries. Those figures describe that competition and should not be treated as a current score for every model or as directly comparable to results from a different evaluation setup.
ARC Prize’s verified-results page labels its OpenAI GPT-6 Astra entry September 2, 2026. On that model-specific page, ARC-AGI-2 scores range from 59.6% with no reasoning to 95.0% with max reasoning across the listed reasoning variants. Read those figures with their model and reasoning labels; they do not establish results for other models or environments. The page also reports different ARC-AGI-3 figures for Standard and Provider Adapter harnesses, illustrating why the harness must be named for interactive results.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What a useful ARC-AGI score report includes
- Benchmark edition and, for ARC-AGI-3, the harness.
- Evaluation split and whether it is public, semi-private, or private.
- Model name, reasoning level, and token limits.
- Scoring rule, output format, and attempt budget.
- Accuracy, cost per task, and total evaluation duration, with the measurement setup where known.
- Verification status and the date of the result.
Keep ARC-AGI-1/2 static-grid results separate from ARC-AGI-3 interactive results. Likewise, do not combine results from different ARC-AGI-3 harnesses or treat a public score as proof of performance on withheld tasks. Benchmark rules, leaderboards, and configurations can change, so a dated result is more informative than an undated ranking.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
