ARC-AGI-2

Overview

ARC-AGI-2 is the current static successor to ARC-AGI. It keeps the same basic format - colored-grid abstraction tasks with a few input/output demonstrations - but rebuilds the benchmark around a harder and better-controlled task distribution.

The motivation is explicit: ARC-AGI-1 inspired useful research, but by 2024 it had become less reliable as a frontier signal. Some tasks were susceptible to brute-force program search, the original private set was small, and the public evaluation set had been exposed for years. ARC-AGI-2 responds by adding harder tasks, first-party human testing, calibrated public/semi-private/private splits, and a lower attempt budget.

At launch, the gap was large. The ARC-AGI-2 paper reported that top systems on the semi-private set were below 5% as of May 14, 2025, while every evaluation task was confirmed solvable by at least two independent non-expert human testers. That picture changed quickly: the official leaderboard generated on 2026-07-23 lists several frontier reasoning systems above 80% on the semi-private axis, with GPT-5.6 Sol Max at 92.5%. ARC-AGI-2 is therefore still important, but its frontier headroom is now visibly narrowing.

Task details

The public repository contains 1,000 training tasks and 120 public evaluation tasks. Training tasks demonstrate the ARC format and priors; public evaluation tasks are intended for testing systems that have not seen those tasks during development.

Each task is a JSON file with demonstration pairs and test pairs. The model sees the demonstration input/output grids and the test input grid, then constructs the missing output grid. Values are integers from 0 to 9, rendered as colors; grids range from 1x1 to 30x30.

ARC-AGI-2 also has two non-public tiers:

Semi-private evaluation: for testing remotely hosted commercial models with low leakage probability.
Fully private evaluation: for self-contained competition systems, with the lowest leakage probability.

ARC Prize states that these tiers are calibrated to similar human-facing difficulty so public, semi-private, and private results are more predictive of one another than ARC-AGI-1’s older split structure.

Example

ARC-AGI-2 examples look deceptively similar to ARC-AGI-1: a few colored-grid transformations and a held-out test grid. The difference is in the rule structure. The technical report highlights tasks that require symbolic interpretation, compositional reasoning, and context-dependent rule application.

A simple ARC-AGI-1 task might be solved by detecting a global symmetry or applying one transformation to every object. ARC-AGI-2 more often requires deciding which symbols are meaningful in context, combining several local rules, or applying a rule only to objects selected by an inferred property. That makes naive exhaustive search and shallow pattern matching less effective.

Scoring methodology

The score is exact task accuracy. A task is solved only if the system produces the correct output grid for every test input. ARC-AGI-2 allows two trials per test input, down from ARC-AGI-1’s three.

There is still no partial credit. A visually plausible grid with one wrong cell is incorrect. That harsh scoring is intentional: the benchmark asks whether the system inferred the task rule well enough to execute it, not whether it got close.

Because systems can use very different search budgets, cost is part of the result. ARC Prize reports model/system labels such as “pure LLM,” “single CoT,” and “CoT + search/synthesis” because those are not equivalent evaluations. The right comparison unit is the full system under a named compute budget.

What it doesn’t measure

ARC-AGI-2 measures static visual-symbolic abstraction. It does not measure real-world tool use, long-horizon planning, open-ended language tasks, factual knowledge, safety behavior, or interactive adaptation. It also does not evaluate whether a model can learn across a long sequence of changing environments.

That last gap is important for lineage: ARC-AGI-3 is the next ARC generation, but it changes the task from static grid completion to interactive agentic learning. For the same static colored-grid format, ARC-AGI-2 remains the successor to ARC-AGI-1.

Known issues & criticisms

Public evaluation is still public. The 120 public evaluation tasks support open research, but repeated development against them will overfit. Use semi-private or fully-private results for high-stakes claims.
The task count is small. With 120 public eval tasks, one task moves the score by 0.83 points. Small differences between systems are not meaningful without repeated, controlled evaluation.
System comparisons are hard. A pure model, a reasoning model with a long chain-of-thought budget, and a search/synthesis stack are different objects. The current leaderboard also shows steep performance changes across reasoning levels, so “model A beats model B” claims are easy to overstate.
Saturation is split-dependent. The semi-private leaderboard crossed 85% in 2026, but Kaggle/private competition evaluation and reproducibility constraints remain separate evidence.
It is still a specialized abstraction format. Strong ARC-AGI-2 performance would be important evidence of fluid reasoning, but it would not settle broader claims about general intelligence.

Contamination notes

Risk: medium for the benchmark as a whole, and high for repeated public-eval tuning. ARC-AGI-2 improves on ARC-AGI-1 with newly curated tasks and private evaluation tiers, but the public repository intentionally exposes the training set and public evaluation set. That is the right tradeoff for open research and the wrong split for definitive frontier claims.

Prefer ARC Prize verified or competition results when you need leakage controls. For local research, keep the public evaluation set untouched until final testing and do not iterate on prompts, programs, or model selection using public-eval feedback.

How to run it

Dataset:

git clone https://github.com/arcprize/ARC-AGI-2
# data/training has 1,000 training tasks
# data/evaluation has 120 public evaluation tasks

Task format:

import json

with open("data/evaluation/<task-id>.json", "r", encoding="utf-8") as f:
    task = json.load(f)

print(task["train"][0]["input"])
print(task["train"][0]["output"])
print(task["test"][0]["input"])

For benchmark reporting, include the split, attempt budget (pass@2), whether the system used search/synthesis/tools, and cost per task. A public-evaluation score from a hand-tuned workflow is not comparable to a private verified score.

History & lineage

2019-11 - ARC-AGI-1 introduces the colored-grid abstraction format.
2024-12 - ARC Prize reports rapid ARC-AGI-1 progress and identifies weaknesses in the original private set, including small size, overfitting risk, and susceptibility to brute-force program search.
2025-03 - ARC-AGI-2 and ARC Prize 2025 are announced. Launch results show leading AI systems below 5% on ARC-AGI-2.
2025-05 - ARC Prize publishes the ARC-AGI-2 technical report, emphasizing calibrated difficulty, first-party human testing, less brute-forceable tasks, and public/semi-private/private evaluation tiers.
2025-12 - NVARC wins the ARC Prize 2025 high-score track with 24.0% on ARC-AGI-2 private evaluation at about $0.20 per task.
2026-07 - The official leaderboard generated on 2026-07-23 lists GPT-5.6 Sol Max at 92.5% on the ARC-AGI-2 semi-private axis.
2026 - ARC-AGI-3 extends the ARC program into interactive environments, changing the evaluation from static answer grids to agentic learning efficiency over time.

shots	few-shot in-task demonstrations
reasoning	agentic
temperature	—
pass@k	pass@2
tools	yes
harnesses	ARC-AGI-2 repository ARC Prize benchmarking

Performance over time