benchwiki
agentic-tool-use

ARC-AGI-3

Active
2026-04-17 → today

Interactive ARC environments that score agents by human-relative action efficiency, planning, exploration, and adaptation.

last updated 2026-07-24

Performance over time

vendor-reported independent
010203040506070809010020252026human baseline 100%

Baseline: The ARC-AGI-3 paper and ARC Prize overview state that environments are 100% human-solvable; the score definition treats 100% as beating every game as efficiently as humans.

Overview

ARC-AGI-3 is the interactive branch of the ARC benchmark series. ARC-AGI-1 and ARC-AGI-2 ask a test taker to infer a static input/output grid transformation. ARC-AGI-3 instead puts an agent inside novel, abstract, turn-based environments. The agent must explore, infer goals, build a working model of the environment dynamics, and plan action sequences without natural-language task instructions.

The benchmark keeps the ARC philosophy: tasks should depend on fluid adaptation and Core Knowledge priors rather than factual knowledge, language fluency, or domain-specific training. The major change is time. ARC-AGI-3 tests whether an agent can learn from experience inside an environment, not just produce a final answer for a static puzzle.

The early gap is large. The ARC-AGI-3 paper reports that humans can solve all environments, while frontier AI systems scored below 1% as of March 2026. The official leaderboard generated on 2026-07-23 shows progress, including GPT-5.6 Sol Max at 7.78% on the semi-private axis, but the benchmark remains far from saturated.

Task details

An ARC-AGI-3 task is an interactive environment made of abstract visual states and discrete actions. The agent receives observations over time, chooses actions, sees the consequences, and continues until it completes the level, fails, resets, or hits a configured budget. There are no pre-loaded natural-language rules; the goal and mechanics must be discovered from interaction.

The paper describes three evaluation sets:

  • Public Demo: 25 environments that demonstrate the format and are intentionally easier and more accessible.
  • Semi-Private: 55 held-out environments for testing models behind external APIs.
  • Fully Private: 55 tightly guarded environments for official competition evaluation.

This means the benchmark has 135 named environments across its public, semi-private, and fully-private tiers, but the public set is a demonstration interface rather than the main evaluation basis.

Example

A typical ARC-AGI-3 environment might present a grid-like game with colored objects, a controllable cursor or entity, and several possible actions. At first, the agent may not know which object is controllable, what counts as progress, or which mechanic changes between levels. It has to probe the environment, remember which actions changed state, infer the hidden objective, and then execute a shorter plan on later levels.

That is different from static ARC. A language model can describe a plausible rule and still fail if it wastes actions, commits to a wrong sub-goal, or does not update its beliefs after new feedback. ARC-AGI-3 rewards efficient embodied experimentation.

Scoring methodology

The primary score is Relative Human Action Efficiency, or RHAE. The benchmark compares an AI test taker’s action count to human action baselines per level, normalizes those comparisons per environment, and aggregates across environments. A score of 100% means an agent can beat every game as efficiently as humans.

Completion alone is not enough. An agent that eventually solves an environment after excessive exploration receives less credit than one that discovers the mechanics and goal efficiently. This is why the official leaderboard emphasizes cost and efficiency as well as score.

The timeline on this page uses official semi-private leaderboard points and converts the leaderboard decimals to percentages. ARC-AGI-3 scores should not be compared directly to ARC-AGI-1 or ARC-AGI-2 exact-match accuracies.

What it doesn’t measure

ARC-AGI-3 does not measure factual knowledge, broad natural-language instruction following, web browsing, tool use in ordinary software environments, or social reasoning. It also does not prove general intelligence by itself. It isolates a specific capability: acquiring a new skill efficiently through interaction in novel abstract environments.

It is also not a simple model-only benchmark. The evaluated object is an agent loop: prompts, memory, observation encoding, action selection, reset policy, and compute budget all matter.

Known issues & criticisms

  • Scores are harder to summarize than exact-match accuracy. RHAE combines completion, action efficiency, human baselines, and environment normalization. The metric is more realistic for agents but less transparent than a static task-success percentage.
  • Public demos are not representative. The paper says the public set is intentionally easier and does not comprehensively cover private mechanics. Public-demo success should not be reported as benchmark success.
  • Budget sensitivity is central. A model can spend thousands of dollars on interactive exploration. The official leaderboard’s score/cost plot is part of the result, not an optional detail.
  • Agent scaffolding matters. Observation formatting, memory compression, action wrappers, and reset strategies can dominate performance. Results should be attributed to full systems, not just base models.
  • The benchmark is young. Evaluation protocols, SDKs, and competition practice may evolve as more agents are tested.

Contamination notes

Risk: medium overall. The public demo environments are exposed and deliberately useful for learning the interface. ARC Prize mitigates this by making the private environments the main basis for evaluation and by designing the public set so it does not comprehensively represent private mechanics.

Because ARC-AGI-3 is interactive, leakage can be subtler than memorizing final answers. Agents can overfit to mechanics, observation encodings, reward patterns, or reset strategies. High-stakes claims should use semi-private or fully-private evaluation with a named prompt, agent harness, budget, and date.

How to run it

Start with the official ARC-AGI-3 page and SDK documentation:

# Open the public game set and integration docs:
# https://arcprize.org/arc-agi/3
# https://docs.arcprize.org

For reporting, include the environment split, total run budget, total cost, action budget, whether the agent used persistent memory or external tools, and whether the score came from public demo, semi-private, or fully-private evaluation.

History & lineage

  • 2019-11 - ARC-AGI-1 introduces static colored-grid abstraction tasks.
  • 2025-03 - ARC-AGI-2 keeps the static grid format but raises difficulty and improves human calibration and private-set controls.
  • 2026-04 - ARC-AGI-3 appears as arXiv:2603.24621v2, introducing interactive turn-based environments and RHAE scoring.
  • 2026-07 - The official leaderboard generated on 2026-07-23 lists GPT-5.6 Sol Max at 7.78% on the ARC-AGI-3 semi-private axis, with substantially higher cost than static ARC runs.

Further reading