benchwiki
maths

FrontierMath

Active
2024-11-07 → today

Original, unpublished research-level math problems by expert mathematicians — mostly private to resist contamination. Frontier models solved under 2% at launch.

last updated 2026-07-22

Performance over time

vendor-reported independent
0102020242025

Baseline: No aggregate human baseline exists, and — as with HLE — none is well-defined in principle, only more so. These are research-level problems spanning many advanced fields; no single mathematician can solve across all of them. The meaningful human reference is per-problem: an expert in the relevant subfield solves it over hours to days. Fields Medalists who reviewed the problems underscored the point — Terence Tao said that in the near term 'the only way to solve them, short of having a real domain expert in the area, is to get a lucky overfit,' and predicted the hardest tier would 'resist AIs for several years.'

Overview

FrontierMath, built by Epoch AI and announced in November 2024, sits at the very top of this wiki’s mathematics difficulty ladder — GSM8K → MATH → AIME → FrontierMath. Where AIME is hard high-school competition math and MATH is hard contest math, FrontierMath is original, unpublished, research-level mathematics: problems written by expert mathematicians that take a specialist in the relevant branch hours to days to solve, spanning number theory, algebraic geometry, combinatorics, analysis, and more. It is the benchmark that currently defines mathematical headroom for AI — and it announced itself with a stark number: at launch, state-of-the-art models solved under 2% of the problems.

The problems are hard in a way credentialed experts vouched for. Fields Medalists reviewed them — Terence Tao (2006), Timothy Gowers (1998), and Richard Borcherds (1998) among them — and Tao’s assessment set the tone: in the near term, he said, “the only way to solve them, short of having a real domain expert in the area, is to get a lucky overfit,” predicting the hardest tier would “resist AIs for several years.” (These attributions are quoted from Epoch’s own materials; see Further reading.)

Two design choices make FrontierMath unusual and complete a story the earlier maths pages set up. First, despite requiring deep reasoning, every problem has an automatically verifiable answer — a specific number or a SymPy-checkable object — so grading is exact-match/equivalence, not human judgment (the same mechanism as MATH, scaled to research problems). Second, and more consequentially, the benchmark is mostly private by design: the problems are unpublished and held out, evaluations are run by Epoch, and only a small sample set is public. That makes FrontierMath the strongest contamination defense on the wiki, and the third point of a taxonomy — MMLU (fully public, fully contaminated) → AIME (fresh yearly, then leaks) → FrontierMath (private held-out) — but it buys that defense with a different cost: you have to trust the evaluator, because you cannot reproduce a private benchmark yourself.

FrontierMath is also the page where this wiki’s neutrality matters most, because the benchmark had a public funding-disclosure controversy (documented in Known issues). A trustworthy reference has to report that plainly and in sourced detail, precisely because most trackers just quote the score.

Task details

FrontierMath is 338 problems in the current version (v2, June 2026), organized into difficulty tiers:

  • Tiers 1–3 (295 problems): the base benchmark — exceptionally hard research-level problems, peer-reviewed by expert mathematicians.
  • Tier 4 (43 problems): a harder expansion added in v2, introduced because frontier progress on Tiers 1–3 had become significant.
  • Open Problems: a separate set of genuinely unsolved research questions (not part of the scored tiers).

Per-tier scores are the meaningful unit — a blended “FrontierMath %” hides which tier, and the tiers differ enormously in difficulty. Because the tiers are small (Tier 4 is 43 problems), per-tier figures carry wide confidence intervals (see the statistical note). On the size convention: this page’s dataset.size is the full current 338; most historical “FrontierMath” scores refer to Tiers 1–3 (295), and the benchmark is versioned (e.g. FrontierMath_11-26-24, FrontierMath_02-28-25), so always cite the version and tier.

  • Input: a research-level problem in a specific advanced field.
  • Output: a numerical or SymPy-checkable answer, verified automatically. No proof is graded — only the final answer (a deliberate design choice enabling automated grading; see What it doesn’t measure).
  • Tools: the standard Epoch evaluation grants models a Python environment and ample compute to experiment and iterate — so the headline configuration is tools-on, which is constitutive here and materially affects scores (see Scoring methodology).

Example

The problems are not reproducible here — the set is private, and only a handful are public — but their character is distinctive. A FrontierMath problem typically asks for a specific computable quantity whose derivation requires genuine research-level machinery: a number-theoretic invariant, the value of a delicate combinatorial sum, a quantity from algebraic geometry. The answer is a concrete number the model must compute correctly, but arriving at it requires the kind of multi-hour reasoning a specialist would deploy — often including writing and running code to explore cases. That combination — deep reasoning to set up, automatic verifiability at the end — is exactly what lets FrontierMath grade research-level math without a human judge, and it is why a Python tool is part of the standard eval.

Scoring methodology

Grading is automated answer verification: each problem has a known numerical/symbolic answer, checked for equivalence (cf. MATH’s equivalence grading, here without an LLM judge). So the scoring mechanism itself is clean. The interpretive hazards are elsewhere, and they are severe:

  • Tools-on vs tools-off, and heavy-compute vs single-pass. The standard eval grants Python and generous compute. A tools-on, high-compute, high-sample run scores far higher than a plain single pass. A “FrontierMath X%” figure is meaningless without the tool and compute setting — this is the HLE tools-on/off trap in its most extreme form, because these problems reward code execution heavily.
  • Announced vs independently-reproduced scores. FrontierMath has a documented gap between vendor-announced numbers and Epoch’s own reproductions. When OpenAI announced o3 at 25.2% (Dec 2024), Epoch’s independent evaluation of a comparable model (o3-mini high) came in around 11%; Epoch noted the higher figures may reflect extreme compute, tool use, scaffolding, or majority voting, and that there was “no publicly accessible way” to reach that level. Treat announced numbers as an upper bound and prefer Epoch’s.
  • Private-set trust. Every score is an Epoch-run measurement on a held-out set. You cannot reproduce it independently — a real methodological cost (see Known issues).

This page plots only Epoch-attributable numbers and labels each with tier and tool setting, precisely because the alternative — quoting whichever number is highest — is how FrontierMath gets misreported.

What it doesn’t measure

FrontierMath measures research-level mathematical problem-solving with a single verifiable answer, tools-on — a narrow slice by design. It is not general reasoning: excelling here says little about non-mathematical capability. Even within mathematics, the automatic-verification requirement excludes proof-based work: FrontierMath problems must have a checkable final answer, so the vast domain of proving theorems — the actual substance of research mathematics — is out of scope. That is the province of proof-graded evaluations (USAMO/IMO proof grading, and emerging automated-proof benchmarks), the “process matters” complement here just as it is for AIME and MATH. And because the headline is tools-on, a FrontierMath score reflects a model plus a Python environment plus compute, not the model alone.

Known issues & criticisms

  • Funding-disclosure controversy (most important). FrontierMath’s creation was funded by OpenAI, which “commissioned 300 questions for the core FrontierMath dataset and 50 problems for Tier 4” (Epoch’s own statement), and OpenAI had access to most of the problems. This funding was not disclosed at the November 2024 announcement; it appeared only in a later version of the paper. Reporting (TechCrunch, 19 Jan 2025) documented that many contributing mathematicians were unaware of OpenAI’s involvement — some were paid roughly $300–$1,000 to write problems without being told who funded the work or would have access — and that this might have affected their decision to participate. The central concern raised was whether OpenAI’s access could have influenced its models’ reported scores. Epoch acknowledged the transparency lapse. In response, safeguards were established: Epoch withheld a held-out set the funder cannot access (“53 solutions withheld from OpenAI” in one version; “20 of 50 Tier 4 problems reserved as holdout”), and states it “retains the right to conduct and publish evaluations using the full FrontierMath dataset.” These specifics are cited to Epoch’s own materials and TechCrunch (see Further reading); this page reports them without editorializing.
  • Trust-the-evaluator (the flip side of privacy). A private benchmark cannot be independently reproduced. Its low contamination risk is bought by requiring trust in Epoch’s evaluation — a genuine methodological cost, distinct from the contamination risk it removes.
  • Tools-on / heavy-compute non-comparability. As in Scoring methodology: tool and compute settings swing scores by tens of points, and announced numbers have exceeded independent reproductions.
  • Tiny per-tier N. Tiers are small (Tier 4: 43 problems), so per-tier percentages are statistically noisy.
  • Narrow by design. Research-level, verifiable-answer math only — not general reasoning, and not proof-based mathematics.

Contamination notes

Risk: low — the first low-risk benchmark on the wiki, and the counter-model to everything before it. The problems are original and unpublished, and the benchmark is mostly private: only a small sample is public, and Epoch evaluates on a held-out set that is never released. Nothing to memorize means near-zero training-data leakage — structurally the opposite of MMLU’s fully-public exposure and stronger even than AIME’s fresh-then-leaks yearly cycle. Together the three complete a contamination-strategy taxonomy: public-and-static (contaminated forever), yearly-refresh (briefly clean), and private-held-out (clean but unreproducible). The honest caveats: “low” is not “none” — the funder had access to most problems, which is exactly why the held-out safeguards described above exist — and privacy converts a contamination risk into an evaluator-trust requirement. Low contamination risk here is real, but it is inseparable from the governance questions in Known issues.

How to run it

You largely cannot run FrontierMath yourself — that is the point. The full benchmark is private; evaluations are conducted by Epoch AI on the held-out set. Practically:

# Canonical results and methodology: https://epoch.ai/frontiermath
# A small set of sample problems is public for inspection:
#   https://epoch.ai/frontiermath/benchmark-problems
# Headline scores are Epoch-run, tools-on (Python), and reported per tier/version.

When citing a FrontierMath number, always state: the version (e.g. FrontierMath_02-28-25), the tier (1–3 vs 4), the tool/compute setting (tools-on is standard; heavy-compute/high-sample inflates), and that it is an Epoch private-set result. A bare “FrontierMath 25%” omits all four and is not interpretable — and, per this benchmark’s history, may be an announced figure that independent evaluation does not reproduce.

History & lineage

  • 2024-11 — Epoch AI announces FrontierMath (arXiv:2411.04872): hundreds of original, unpublished research-level problems, automatically verifiable, mostly private. State-of-the-art models solve under 2% — the headline that frontier systems almost entirely fail. Fields Medalists vouch for the difficulty.
  • 2024-12 — OpenAI announces o3 at 25.2% on FrontierMath (tools-on) — a dramatic jump that immediately raises interpretation questions, since Epoch’s independent reproductions of comparable models are far lower (~11%).
  • 2025-01 — The funding-disclosure controversy breaks (TechCrunch, 19 Jan 2025): OpenAI’s funding and problem access, undisclosed at announcement, become public; contributing mathematicians report being unaware. Epoch acknowledges the lapse and details held-out safeguards.
  • 2025 — Reasoning models with tools continue to climb on Tiers 1–3; reported figures rise but remain tool/compute-dependent and contested.
  • 2026-06 — Epoch releases v2 with a harder Tier 4 (total 338 problems), added because Tiers 1–3 progress had become significant — a signal that the frontier is advancing even as the hardest tier holds.

FrontierMath has no successor (successor is null): it is the current top of the maths headroom chain. Its complement, not its successor, is proof-graded evaluation (USAMO/IMO), which tests the theorem-proving that FrontierMath’s verifiable-answer format deliberately excludes.

Further reading