Overview
MathArena is an evaluation platform rather than one frozen test file. Its central idea is to run models on recurring math competitions as soon as organizers release the problems. This creates a short clean window in which the questions could not have appeared in the evaluated models’ training data. The platform publishes raw outputs, per-problem results, token use, and cost.
The scope has expanded beyond final-answer contests to proof-based olympiads, visual math, Project Euler, recent-arXiv research questions, false-statement refusal, and formalized Lean statements. That breadth makes MathArena one of the most useful current math trackers, but its top-level expected-performance number changes whenever models or benchmark membership change.
Task details
Final-answer tracks include current AIME, HMMT, SMT, and related contests. Proof tracks include USAMO, IMO, Putnam, and other university or olympiad competitions. Newer platform sections include ArXivMath, BrokenArXiv, ArXivLean, Visual Math, and Project Euler. The official runner normally samples each model four times per problem.
Example
The repository demonstrates a complete AIME 2025 run:
uv run python scripts/run.py --comp aime/aime_2025 --models openai/gpt-4oThe site exposes each raw response behind its problem cell, making answer extraction and judging auditable in a way aggregate-only leaderboards are not.
Scoring methodology
Competition pages retain their natural metrics: exact answers for AIME-like exams and rubric points for proofs. The cross-platform expected performance summary fits item difficulty and discrimination parameters, predicts correctness for every current non-deprecated question, and averages those predictions. It is a model-based index, not raw percent correct. Cost is averaged separately over the current problem mix.
What it doesn’t measure
MathArena focuses on contest and research-style mathematics. It does not measure ordinary quantitative literacy, classroom teaching, formal proof correctness unless a formal track is selected, or stable year-over-year performance on a fixed population of items.
Known issues & criticisms
Real-time evaluation reduces contamination only during a time window; public problems can enter later training data. Competition sets are small, so four samples do not eliminate variance. Proof scores depend on a judging pipeline and rubric, while cross-platform IRT aggregates depend on which competitions are marked non-deprecated. Cost comparisons also inherit provider pricing and token-accounting differences.
Contamination notes
Risk is low at first evaluation, not permanently low. The design evaluates newly published problems immediately and continuously adds replacements. Cite the competition and evaluation date. Re-running an old MathArena set years later is a public-test evaluation with a different contamination risk.
How to run it
Use the official repository:
git clone https://github.com/eth-sri/matharena.git
cd matharena
python -m pip install -e .
python scripts/run.py --comp aime/aime_2025 --models openai/gpt-4o --n 4Competition and model arguments refer to YAML files under configs. Proof and false-statement configurations may require a separate scripts/judge/judge.py pass. Pin the commit and store raw generations.
History & lineage
The first paper appeared on 2025-05-29 with seven competitions and 162 problems. MathArena then grew into a continuously maintained platform with many competition seasons and new research/verification tracks. It complements AIME by providing clean annual snapshots rather than treating one public year as timeless.
Further reading
- 2025 benchmark paper: https://arxiv.org/abs/2505.23281
- 2026 platform paper: https://arxiv.org/abs/2605.00674
- Official live leaderboard: https://matharena.ai
- Official evaluation code: https://github.com/eth-sri/matharena
- Datasets and raw model outputs: https://huggingface.co/MathArena