benchwiki
maths

Omni-MATH

Deprecated
2024-09-20 → today

A 4,428-problem Olympiad mathematics suite spanning more than 33 subdomains and ten annotated difficulty levels.

last updated 2026-07-29

Performance over time

vendor-reported independent
55652024

Baseline: No aggregate human baseline was published across the 33+ domains and ten difficulty levels.

Overview

Omni-MATH assembled 4,428 English, text-only Olympiad-level problems into a broad taxonomy of more than 33 subdomains and ten difficulty levels. It was designed after MATH and GSM8K had become too easy for frontier systems, offering a much larger contest-math set than annual exams such as AIME.

The content remains useful for analysis, but the original leaderboard metric is no longer reliable enough for frontier ranking. In 2026, Omni-MATH-2 reported an expert audit of judge disagreements and found that the original Omni-Judge was wrong in 96.4% of those cases. This page therefore treats original Omni-MATH as deprecated for comparative claims, not as worthless data.

Task details

Each item contains a competition problem, reference solution/final answer, domain labels, and difficulty metadata. The model generates a worked response. The original protocol extracts its final answer and asks GPT-4o, or the cheaper Omni-Judge model, whether it is mathematically equivalent to the reference.

Example

The official repository’s judge example compares the submitted expression 2017^98 with the reference answer and asks for a binary equivalence verdict plus justification. This shows why string exact match is insufficient, but also why judge quality becomes part of the measurement.

Scoring methodology

The original leaderboard uses GPT-4o to judge final-answer equivalence and averages binary verdicts. Omni-Judge was trained to imitate these verdicts, with the repository reporting roughly 95% agreement on a 2,690-query internal set. Agreement with GPT-4o is not agreement with mathematical truth; the later expert audit demonstrated that both extraction and judging can dominate the observed score.

What it doesn’t measure

Omni-MATH grades final answers, not proof validity or research mathematics. It is English-only and text-only. A correct final expression can receive credit despite invalid reasoning, while a correct derivation can lose credit through parsing or judge error.

Known issues & criticisms

The decisive issue is evaluator saturation. Benchmarks Saturate When The Model Gets Smarter Than The Judge introduced Omni-MATH-2, separating 4,181 clean exact-answer items from 247 non-standard items and reporting that Omni-Judge caused nearly all audited disagreements. The original public set also includes competition material vulnerable to contamination.

Contamination notes

Risk is high. All 4,428 problems and answers are public, and their competition sources may appear elsewhere online. Domain and difficulty annotation do not mitigate exposure. Use fresh or private problems when evaluating generalization.

How to run it

The historical evaluator is reproducible from the official repository:

git clone https://github.com/KbsdJames/Omni-MATH.git
cd Omni-MATH/GPT_eval
bash get_result.sh

For new work, prefer the clean exact-answer portion defined by Omni-MATH-2 and a verified symbolic/rule-based checker. If reproducing an old result, pin the judge model snapshot, prompt, parser, and repository commit.

History & lineage

The dataset was released on 2024-09-20, its paper followed on 2024-10-10, and rule-based evaluation code was added in December 2024. Omni-MATH-2 is a methodological revision of the same 4,428 items, not a directly comparable new score axis; it is described in prose because no local omni-math-2 page exists yet.

Further reading