AIME

Overview

AIME is different from every benchmark before it on this wiki: it is not a research dataset but a real human competition exam. The American Invitational Mathematics Examination has been administered by the Mathematical Association of America since 1983 as the second stage of the path to the USA Math Olympiad — you qualify by scoring in roughly the top 5% of the AMC 10/12. There is no paper, no arXiv, no research authors; there is a problem committee, an exam, and a few hundred thousand of the country’s strongest young mathematicians sitting it each February.

That origin makes AIME a natural, brutal benchmark for machine mathematical reasoning, and it neatly sidesteps two problems that dog the research datasets. First, grading is trivial: every AIME answer is an integer from 000 to 999, so scoring is pure exact-match — none of the mathematical-equivalence checking that makes MATH grading contentious (½ = 0.5, equivalent algebraic forms). An integer either matches or it doesn’t. Second, and more importantly, AIME is refreshed every year: a brand-new exam appears each February, briefly uncontaminated because it did not exist when the models were trained. This is the concrete answer to the contamination problem the GSM8K and MATH pages keep pointing at — the difficulty ladder GSM8K → MATH → AIME → FrontierMath is also a ladder of contamination defenses, and AIME is where “refresh the test” first appears.

The results that made AIME the reasoning-era’s favorite math benchmark are stark. On AIME 2024, GPT-4o scored 13.4%; then reasoning models arrived and o1 hit 74.4%, o3 96.7% — a near-solve in a single year. What makes those numbers striking is the human context: AIME takers are already elite, and even they median about 5 out of 15 (≈33%). A model clearing 90% is beating the vast majority of the strongest young mathematicians in the United States, on their own exam.

AIME’s status is genuinely both active and saturated, for structural reasons — which is why this page calls it nearing-saturation and spends real words on the tension. Old exams are solved and contaminated; each fresh exam has a shrinking clean window; frontier models score high even on the fresh ones. The benchmark survives by refreshing faster than it saturates — but the margin is thin.

Task details

AIME is a real exam with a fixed structure:

Two exams per year: AIME I and AIME II, 15 questions each, so 30 questions per year.
Answers are integers 000–999. No multiple choice, no free-form expressions — just a three-digit integer, which makes grading pure exact-match.
No calculator on the real exam. This makes tools: false constitutive, not a setting: a code-interpreter or tool-augmented run measures a different capability (can the model orchestrate a calculator) than the closed-book reasoning AIME is meant to test. Report tool-augmented AIME numbers as a separate thing.
Input: a competition problem; the model answers zero-shot with chain-of-thought.
Output: an integer answer, graded by exact match.

dataset.size is 30 — a full exam-year (AIME I + II), the common reported configuration — following the size-matches-primary convention (see GPQA); a single exam is 15. The problems are published by the MAA and compiled by the community (MathArena, per-year HuggingFace sets), but they are MAA copyright — the first benchmark here whose data is not cleanly open-licensed (contrast the MIT-licensed research datasets). Redistribution is restricted; evaluate via the published problems, and mind the licensing when mirroring them.

Example

AIME problems are hard in a specific, contest way — a clever observation collapses a seemingly heavy computation:

Problem (AIME-style): Let (N) be the number of ordered pairs ((a,b)) of positive integers with (a+b \le 100) and (\tfrac{a}{b}) an integer. Find (N \bmod 1000).

Answer: a specific integer 000–999 (found by counting, for each (b), the multiples of (b) that keep (a+b \le 100), then summing).

The integer-answer format is the whole trick: the problem may require a page of casework, but the deliverable is one number that is either right or wrong. That is what makes AIME cleanly gradable and, at the same time, unforgiving — a single arithmetic slip anywhere in the derivation yields a wrong integer and zero credit, exactly as for a human sitting the exam.

Scoring methodology

Scoring is exact-match on an integer — the simplest grading on the wiki, with no equivalence checker to argue about. The hard problem here is sampling convention, and it is the sharpest reporting trap in the whole maths capability.

Because an exam is only 15 questions, a single pass is statistically fragile (see the statistical note). So labs almost never report a single pass@1 run. Instead:

avg@k — run the model k times per question, average the pass@1 scores (MathArena uses avg@4).
maj@k / cons@k — take the majority-vote answer over k samples (e.g. maj@64); this is much higher than a single pass because the model gets many tries to converge.
pass@1 — a single sample; the lowest and noisiest number.

These differ by double digits on the same model. A model reported at “90% on AIME 2025” under maj@64 might be ~80% at avg@1. So an AIME number is meaningless without its sampling convention and its exam year. Two “AIME” figures are comparable only if both match. This page’s plotted timeline fixes both — AIME 2024, pass@1 — precisely so the points mean one thing.

Schema note (flagged, not acted on): the pass_at field is a single integer and cannot distinguish avg@64 from maj@64 from pass@1-with-averaging; and timeline points have no structured place to record exam-year or sampling, so this page encodes both in each point’s model label and source. AIME, plus future yearly benchmarks (AMC, MathArena events, window-based code evals), may finally justify adding structured exam_year + sampling fields to timeline points — but not for one page.

What it doesn’t measure

AIME measures fluency at contest mathematics — the specific genre of clever, integer-answer competition problems — under closed-book, no-tool conditions. It is not research mathematics: no proofs, no novel results, no open problems (that is FrontierMath and USAMO/proof-grading evals). It does not measure the validity of reasoning: only the final integer is graded, so a right answer from wrong working scores full marks, and a correct method with one slip scores zero — the same limitation as GSM8K and MATH, and the reason proof-graded benchmarks (USAMO evaluations) exist as the “process matters” successor idea. The integer format also gives a small nonzero lucky-guess floor. And AIME rewards competition-specific pattern fluency, which is real but narrow: a high AIME score does not imply general or research mathematical ability. For that, climb to FrontierMath.

Known issues & criticisms

Sampling-convention non-comparability (first-order). As above: avg@k vs maj@64 vs pass@1 swing the same model by double digits. This is the single biggest source of bogus AIME comparisons; always demand the convention.
Tiny-N statistical fragility. 15 questions per exam, 30 per year — one question is 6.7 / 3.3 points. Small gaps are noise (see statistical note). This is why heavy sampling exists, and why per-exam claims need care.
Contamination decays within each cycle. A fresh exam is clean for weeks to months, then floods the web with solutions. Any AIME-2024 number today is memorization-inclusive; a current-year number is only briefly trustworthy (see Contamination).
Licensing / redistribution murk. AIME problems are MAA copyright, not openly licensed, so community datasets sit in a grayer legal area than the MIT research sets — mind redistribution.
Integer-answer format. A nonzero guess floor, and final-answer-only grading that masks reasoning errors (cross-link MATH).
Scope. AIME is contest math, not general or research math; a strong score is easily over-read.

Contamination notes

Risk: medium — and AIME is the page where the wiki’s contamination story finally flips. Every static benchmark so far (MMLU, GSM8K, MATH) degraded monotonically: public once, contaminated forever. AIME is the first where refresh is the defense. Each new exam is a genuine, never-before-seen test at release, so it offers a brief clean measurement — the concrete realization of the yearly-refresh strategy the GSM8K and MATH pages gesture at. But the window is short: within months the problems and worked solutions are everywhere, and that exam-year joins the contaminated pile. So AIME 2024 is now high-contamination while a current-year exam is briefly clean — the risk is a rolling function of exam age, not a fixed level. “Medium” captures the average reality: there is always a fresh exam to run (so never as hopeless as a frozen public set), but no exam stays clean for long (so never truly low-risk). The lesson generalizes to every by-year benchmark (AMC, MathArena’s rolling competitions), and it is why year-fresh evaluation is the current best practice for math.

How to run it

MathArena is the canonical live tracker — it evaluates each new competition on release (avg@k), which is the closest thing to a trustworthy AIME number because it captures models during the clean window:

# Browse / cite results at https://matharena.ai (AIME 2024, 2025, 2026, …)
# MathArena runs each model multiple times per problem and reports the average.

lm-evaluation-harness (per-year AIME tasks exist; specify the year and sampling):

pip install lm-eval
lm_eval --model hf \
  --model_args pretrained=<model> \
  --tasks aime_2024 \
  --batch_size auto
# ALWAYS report: exam year, sampling convention (pass@1 / avg@k / maj@64), tools off

A bare “AIME 92” is uninterpretable: it omits the exam year (which test?) and the sampling convention (which could move it 15+ points). State both, or the number means nothing.

History & lineage

1983 — The MAA introduces AIME as the invitational round between the AMC and the USAMO. For four decades it is a human competition, not a machine benchmark.
2024-02 — AIME 2024 runs. As reasoning models emerge, labs adopt it as a standard math benchmark: GPT-4o scores 13.4%, establishing the pre-reasoning floor.
2024-09 → 2024-12 — The reasoning-model surge: o1 reaches 74.4% and o3 96.7% on AIME 2024 (pass@1) — a near-solve within the year, and the plotted timeline on this page.
2025 — AIME 2025 is fresh at release; frontier models still score high (~85–91%: Grok 4 90.6, o3-mini 86.5, Gemini 2.5 Pro 85.8), showing the yearly refresh slows but does not stop saturation. MathArena becomes the canonical rolling tracker.
2026 — AIME 2026 is the newest clean exam; the cycle repeats. The benchmark persists by refreshing, not by getting harder.

AIME has no successor (successor is null): its headroom heirs in spirit are FrontierMath (research-level) and proof-graded olympiad evals (USAMO), but those are separate lineages, not replacements. The ladder to climb for harder math is GSM8K → MATH → AIME → FrontierMath.

Performance over time