benchwiki
reasoning

ARC-AGI

Saturated
2019-11-05 → today

Colored-grid abstraction tasks that test few-shot rule induction, not memorized facts, language fluency, or school math.

last updated 2026-07-24

Performance over time

vendor-reported independent
0102030405060708090100202420252026human baseline 98%

Baseline: The official ARC Prize leaderboard lists a Human Panel score of 98% on the ARC-AGI-1 semi-private axis. Older public-page summaries also report lower average-human estimates; the sidebar uses the current official leaderboard value.

Overview

ARC-AGI, originally the Abstraction and Reasoning Corpus, is a benchmark for few-shot abstract rule induction. Each task is a tiny visual puzzle made of colored grids. The model sees a handful of input/output examples, then must infer the transformation rule and draw the output grid for a new input. The grids are symbolic rather than photographic: each cell is an integer from 0 to 9, usually visualized as a color.

The important design choice is what ARC leaves out. It is not a trivia test, not a school-maths exam, and not a language-understanding benchmark. Most tasks need only elementary priors like objectness, counting, symmetry, continuity, containment, and simple geometry. The intended capability is closer to fluid abstraction: can a system adapt to a new task from a few demonstrations without being told the rule?

ARC-AGI-1 is now saturated for frontier evaluation. That does not mean models solve it cheaply or cleanly; it means the original benchmark no longer gives the kind of durable frontier signal it was built for. The public evaluation set has been online since 2019, the private sets are small and heavily exercised, and the official ARC Prize leaderboard generated on 2026-07-23 lists several systems above 95% on the ARC-AGI-1 semi-private axis. For current static ARC-style evaluation, use ARC-AGI-2; for interactive agentic evaluation, see ARC-AGI-3.

Task details

An ARC task is a JSON file with two fields:

  • train: demonstration input/output pairs, typically three examples.
  • test: one or more input grids whose output grids must be produced.

Each grid is a rectangular matrix of integers from 0 to 9. Grids can be as small as 1x1 and as large as 30x30. Solving a task means producing the exact output grid for every test input: correct height, width, and every cell value.

The evaluation split has 400 public tasks. The repository also includes 400 training tasks used for prototyping algorithms or learning ARC-relevant priors. ARC Prize leaderboards and competitions have used private or semi-private evaluation sets alongside the public set; scores across those splits should not be collapsed into one undifferentiated “ARC score.”

Example

A typical task might show three demonstrations where a colored shape is copied, all holes inside it are filled with a second color, and unrelated noise pixels are removed. The test input then contains a new shape with a different size and position. A correct system must infer the abstract operation from the examples and construct the exact target grid, not describe the rule in words.

That is why ordinary language ability helps only indirectly. A model can explain “fill the enclosed region” and still fail if it chooses the wrong output size, misses one cell, or applies the rule to the wrong object. ARC rewards executable abstraction.

Scoring methodology

The headline score is task accuracy: the percentage of evaluation tasks solved. There is no partial credit for getting most cells right. If a system predicts a 9x9 grid where the answer is 10x9, or gets one colored cell wrong, that test output is incorrect.

ARC-AGI-1 allows three trials per test input. This makes reporting sensitive to search and refinement loops: a system that samples many candidate programs or grids, ranks them, and submits its best attempts is not comparable to a single forward pass from a base model. ARC Prize reporting therefore increasingly pairs accuracy with efficiency, usually cost per task or a fixed compute budget.

What it doesn’t measure

ARC-AGI does not measure broad knowledge, multilingual competence, long-context synthesis, tool use in realistic software environments, or natural-language instruction following. It also does not prove general intelligence by itself. The tasks deliberately isolate one narrow but important behavior: adapting to a new symbolic task from a few examples.

It is also not a pure “LLM benchmark” anymore. Strong ARC systems often combine a model with program synthesis, search, self-refinement, test-time training, or domain-specific heuristics. That is legitimate, but it changes the unit being evaluated from “model” to “system.”

Known issues & criticisms

  • Public-set exposure is unavoidable. The 400 public evaluation tasks have been online for years. A public ARC-AGI-1 score from a frontier model should be treated as exposure-inclusive unless the evaluator proves otherwise.
  • Compute changes the meaning of a score. ARC can reward large test-time search. The o3-preview results were important because they showed strong adaptation, but ARC Prize also reported very high costs and sample counts for the high-compute setting.
  • Private sets are small. ARC Prize has noted that the original private evaluation set has only 100 tasks and has seen many submissions over time, raising overfitting risk.
  • Brute-forceability became a benchmark problem. Program-search approaches solved a significant fraction of ARC-AGI-1 tasks. That is part of why ARC-AGI-2 was created.
  • Single-number reporting hides split differences. Public, semi-private, and private scores can diverge, and competition systems may run under very different constraints from commercial model evaluations.

Contamination notes

Risk: high for the public evaluation set. The task files, example interfaces, writeups, and solutions have circulated since 2019. Even without direct memorization, teams and model trainers can tune against the task distribution.

ARC Prize mitigated this with semi-private and private sets for verified testing, plus explicit policies around cost and reproducibility. Those mitigations are external to the original ARC-AGI-1 public dataset. If contamination resistance matters, prefer ARC-AGI-2’s newer public/semi-private/private structure or ARC-AGI-3’s private interactive environments, and use the ARC Prize verified testing policy.

How to run it

Dataset and browser interface:

git clone https://github.com/fchollet/ARC-AGI
# data/training has 400 training tasks
# data/evaluation has 400 public evaluation tasks
# apps/testing_interface.html can be opened locally to inspect tasks

ARC Prize benchmarking harness:

git clone https://github.com/arcprize/arc-agi-benchmarking
# follow the repository instructions for model adapters and ARC-AGI-1/2 runs

Always report the split, attempt budget, whether the system used tools/search, and approximate cost. A bare “ARC-AGI 75” is not enough information to compare systems.

History & lineage

  • 2019-11 - Francois Chollet releases “On the Measure of Intelligence” and ARC, framing intelligence as skill-acquisition efficiency rather than memorized task performance.
  • 2020 - The original Kaggle competition establishes ARC as a difficult program-synthesis and abstraction benchmark.
  • 2024-06 - ARC Prize launches an official public leaderboard and new competition infrastructure around ARC-AGI-1.
  • 2024-12 - ARC Prize reports major progress: competition systems reach the mid-50s on private evaluation, and o3-preview-low reaches 75.7% on the semi-private leaderboard at the stated public-leaderboard compute limit.
  • 2025-03 - ARC Prize announces ARC-AGI-2 as the harder successor, keeping the static colored-grid format but improving difficulty calibration and private-set controls.
  • 2026-07 - The official leaderboard generated on 2026-07-23 shows ARC-AGI-1 above 95% for multiple verified systems, with GPT-5.6 Sol xHigh listed at 97.5% on the semi-private axis.

Further reading