MATH

Overview

MATH is the benchmark that asked whether language models could do real competition mathematics — not grade-school word problems, but the AMC and AIME-level problems that select for mathematical talent. Introduced by Hendrycks et al. in March 2021, its 12,500 problems come with full worked solutions and span seven subjects and five difficulty levels. Its founding result was a splash of cold water: at a time when scaling was quietly solving most text benchmarks, MATH stayed hard, with the largest models of 2021 scoring single digits (GPT-3 around 5%). The paper’s blunt framing — scaling was “not currently solving MATH” — made it the standard evidence that mathematical reasoning was a distinct, unsolved capability.

MATH is the sibling of MMLU: both came from Dan Hendrycks’s group in 2020–2021, and together they defined how the field measured knowledge (MMLU) and mathematical reasoning (MATH) for years. Where GSM8K tests grade-school arithmetic, MATH tests the next rung up — the same difficulty ladder this wiki traces as GSM8K → MATH → AIME → FrontierMath.

Today almost nobody evaluates on the full 5,000-problem test set. The de facto standard is MATH-500, a 500-problem subset introduced by OpenAI’s Lightman et al. in “Let’s Verify Step by Step” (2023). It exists for two reasons, both worth understanding: (a) grading 5,000 competition problems is slow and expensive, and (b) that paper released PRM800K, a large process-supervision dataset with step-level labels on MATH training problems — so evaluating on a fixed 500-problem subset (representative of the full test distribution) keeps the evaluation clean and cheap. MATH-500 became the number labs actually report, which is why this page treats it as the headline configuration.

MATH is now saturated — but it held out notably longer than GSM8K. GPT-4 managed only ~42% on the full set in 2023, so MATH was still discriminating when GSM8K was nearly solved. It was the reasoning-model era that ended it: o1 reached ~94.8% on MATH-500 in 2024, and frontier models now cluster in the 90s. As always on this wiki, saturated does not mean solved — exact-answer scoring credits the right number regardless of the reasoning, and the honest headroom has moved to harder, fresher math.

Task details

MATH is 12,500 problems — 7,500 train / 5,000 test — sourced from AMC 10, AMC 12, AIME, and similar competitions. Each problem has a full natural-language solution ending in a LaTeX \boxed{} final answer.

Seven subjects: Prealgebra, Algebra, Number Theory, Counting & Probability, Geometry, Intermediate Algebra, Precalculus.
Five difficulty levels: 1 (easiest) to 5 (hardest), enabling per-level and per-subject analysis — though any single level×subject cell is small, so treat fine-grained splits with the CI caution in the statistical note.
Input: a competition problem (in the standard setup, answered zero-shot with chain-of-thought).
Output: a worked solution ending in \boxed{answer}; only the boxed answer is graded.

MATH-500 is a 500-problem subset of the test set, sampled to match the full set’s subject and difficulty distribution. Per the wiki’s size-matches-primary-configuration convention (established on GPQA, whose dataset.size is its 198-problem Diamond subset), this page’s dataset.size is 500 — the headline reported configuration — with the full 7,500/5,000 split recorded here in Task details. GPQA and MATH follow the same rule.

Example

A representative Level-5 Algebra problem:

Problem: Let (f(x) = x^2 - 2x + 3). Find the sum of all values of (x) for which (f(f(x)) = 3).

Solution (sketch): (f(f(x)) = 3) requires (f(x)^2 - 2f(x) + 3 = 3), i.e. (f(x)(f(x)-2)=0), so (f(x)=0) or (f(x)=2). Solving each quadratic and summing the roots (via Vieta’s) gives the boxed total.

Answer: (\boxed{6})

The point is that MATH problems require chained reasoning — substitution, factoring, case analysis, then a final arithmetic step — where a single misstep anywhere yields the wrong boxed answer. This is why chain-of-thought is decisive here (as on GSM8K and BIG-Bench Hard): the model must externalize a multi-stage derivation, not recall a fact.

Scoring methodology

MATH is scored as exact-match on the boxed final answer — but this is the wiki’s first “exact-match” that is emphatically not string equality. The answers are mathematical objects, so grading is by mathematical equivalence: (\tfrac12), (0.5), and (0.500) are the same answer; (\tfrac{1}{\sqrt2}) and (\tfrac{\sqrt2}{2}) are the same; (x+1) and (1+x) are the same. The original release used a normalization script; modern harnesses use sympy-based symbolic equivalence or, increasingly, an LLM judge to decide equivalence.

This makes the checker a first-order source of disagreement:

Harness checkers disagree on edge cases. Different normalizers handle formatting (units, (\pi) vs 3.14, matrix/interval notation, unsimplified radicals) differently, so the same model answers can score a few points apart across harnesses. A MATH-500 number is only comparable within a fixed checker — report which one.
LLM-judge grading reintroduces judge provenance. Using a model to check equivalence inherits every caveat from SimpleQA: the judge can be wrong, and its version drifts, so scores graded by different judges are not strictly comparable.
Answer-only scoring masks flawed reasoning. As on GSM8K, a correct boxed answer from invalid working scores full marks. The process-supervision line of work — PRM800K and “Let’s Verify Step by Step” — exists precisely to grade the reasoning, not just the answer.

Schema note (flagged, not acted on): MATH is the first benchmark where exact-match really means “mathematical-equivalence match,” which is a genuinely different grading mechanism from string exact-match. A dedicated math-equivalence scoring value may be worth adding — but not for one page; AIME, AMC, and Omni-MATH will re-raise this, and the enum should be extended when the second or third such benchmark lands, not now.

What it doesn’t measure

MATH measures competition-mathematics problem solving up to AIME level, with a single checkable final answer — and stops there. It is not research mathematics: no proofs, no open problems, no novel results, which is the domain of FrontierMath. It does not measure the validity of reasoning (only the boxed answer is graded), tool-assisted math (it is closed-book by convention), or reliability under perturbation. It is English-only. And now that it is saturated, a high MATH-500 score says little about frontier models beyond “competent at contest problems it has very likely seen.” For genuine mathematical headroom, climb to AIME (yearly, harder) or FrontierMath (research-level).

Known issues & criticisms

Equivalence-checker disagreement. This is the first-order comparability problem: because answers are graded by mathematical equivalence, different harnesses’ checkers (normalization scripts vs sympy vs LLM judge) disagree on edge cases, moving scores a few points for the same outputs. Always report the checker.
Contamination via PRM800K and web solutions. Public since 2021 with full solutions, and with step-level training-problem solutions released as PRM800K, MATH is heavily represented in training data (see Contamination). Near-ceiling scores are memorization-inclusive.
Exact-answer scoring masks reasoning errors. Same limitation as GSM8K: right-answer/wrong-reasoning scores full marks. Process supervision is the response, not a property of MATH itself.
Competition scope. MATH is contest math, not research math; a strong score is easily over-read as “does mathematics.” Point to FrontierMath for the harder claim.
Small level/subject cells. The 7×5 grid is great for analysis in aggregate but too sparse per cell for reliable per-cell model comparison.

Contamination notes

Risk: high. MATH has every exposure GSM8K has — public since 2021, MIT-licensed, worked solutions all over the web — plus one more: the 2023 PRM800K release circulated step-level solutions to MATH training problems as process-supervision data. That release was scientifically valuable (it underpins modern process-reward models and, ultimately, reasoning models), but it also put more of the benchmark’s solution structure into the open. MATH-500’s motivation is partly entangled with this: sampling a fixed 500-problem test subset both makes grading tractable and keeps evaluation off the train/PRM problems — though the MATH-500 items themselves are public and static, so they contaminate over time like any fixed set. There is no refresh cycle. Treat modern near-ceiling MATH-500 scores as an upper bound on genuine problem-solving, and prefer a year-fresh benchmark (AIME by year) when contamination matters.

How to run it

lm-evaluation-harness (chain-of-thought is the meaningful setting; minerva_math uses the common CoT prompt):

pip install lm-eval
lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-3.1-8B \
  --tasks minerva_math \
  --batch_size auto

MATH-500 (the standard subset), via the PRM800K repo:

git clone https://github.com/openai/prm800k
# math_splits/ contains the 500-problem test subset used as "MATH-500"
# grade with a sympy-based equivalence checker; report which checker you used

Report shots, that CoT was used, and — crucially — the equivalence checker, since checker choice moves the number. State whether you evaluated MATH-500 or the full 5,000-problem set; they are not interchangeable.

History & lineage

2021-03 — MATH released (Hendrycks et al., arXiv:2103.03874; NeurIPS 2021 D&B), sibling to MMLU. Largest 2021 models score single digits; MATH becomes the standard evidence that math reasoning is unsolved.
2022 — Google’s Minerva (a math-specialized model) is the first big jump on the full set, showing targeted training and chain-of-thought move the needle far more than scale alone (discussed here rather than plotted: its headline used majority-vote, not pass@1).
2023 — GPT-4 reaches ~42% on full MATH — strong for a general model, but MATH is still far from solved. OpenAI’s “Let’s Verify Step by Step” introduces PRM800K (process supervision) and the MATH-500 evaluation subset, which becomes the reported standard.
2024 — Reasoning models end it: o1 hits ~94.8% on MATH-500; MATH saturates for frontier evaluation.
2025–2026 — Frontier and reasoning models cluster in the 90s on MATH-500; the discriminating math signal has moved to AIME and FrontierMath.

MATH has no single successor (successor is null): AIME (yearly, harder, contamination-resistant by construction) and FrontierMath (research-level) are the headroom benchmarks, but they are separate lineages, not designated heirs. The difficulty ladder to climb is GSM8K → MATH → AIME → FrontierMath.

Performance over time