benchwiki
maths

OlympiadBench

Active
2024-02-16 → today

Bilingual Olympiad and Gaokao mathematics and physics problems, including diagrams, open answers, and proofs.

last updated 2026-07-29

Performance over time

vendor-reported independent
1020302024

Baseline: No single aggregate human baseline is published across competition, Gaokao, math, physics, text, image, open-answer, and proof subsets.

Overview

OlympiadBench is a bilingual, multimodal science benchmark built from international and Chinese Olympiads plus China’s College Entrance Examination (Gaokao). It contains 8,476 mathematics and physics problems in English and Chinese, including text-only questions, diagram-dependent questions, machine-checkable open answers, and theorem proofs.

Its breadth is the point. A single result averages across mathematical reasoning, physical modeling, bilingual comprehension, visual interpretation, and answer formatting. That makes the suite harder and more realistic than text-only contest sets, but also makes the headline score less diagnostic unless accompanied by subset results.

Task details

Files encode OE (open-ended) or TP (theorem proof), MM (multimodal) or TO (text-only), subject, language, and source family. Each record includes the question, optional context and image placeholders, worked solution, final answer, answer type, and metadata.

Example

The official repository shows an English geometry problem about a snail moving clockwise or counterclockwise on a unit-circumference circle. The required output is the largest constant satisfying a universal avoidance property, with gold final answer 1/2 and a proof. This is representative of the benchmark’s mix of concise final answers and substantial reasoning.

Scoring methodology

Open-ended questions use standardized answer formatting and automated equivalence checks. Theorem-proof questions cannot be reduced to exact final-answer match and were assessed separately or by sampling in the original work. Always specify whether a result covers the full suite, text-only subset, open-ended subset, or a subject/language slice.

What it doesn’t measure

OlympiadBench does not represent everyday quantitative reasoning, research mathematics, interactive tool use, or laboratory science. It also cannot isolate pure mathematical reasoning when images, translation, physics knowledge, and OCR all affect the same aggregate.

Known issues & criticisms

The suite is sourced from existing examinations rather than newly authored private problems, so exposure may predate the release. OCR and manual correction improve quality but can still leave parsing artifacts. Proof scoring is less scalable and less directly reproducible than final-answer grading. The July 2024 dataset update also means results must identify the dataset revision.

Contamination notes

Risk is high. Competition and Gaokao problems originated in public PDFs, and the cleaned benchmark is fully downloadable with solutions. Deduplication removes internal repeats but cannot remove pretraining exposure to the original exams.

How to run it

Use the official evaluator and pin a commit:

git clone https://github.com/OpenBMB/OlympiadBench.git
cd OlympiadBench
# Follow inference/ for generation and eval/ for answer verification.

Record dataset version, selected files, image model and resolution, prompt, decoding, proof-grading policy, and aggregation. Do not compare the repository’s text-only table directly with its full-benchmark table.

History & lineage

OlympiadBench was released on 2024-02-16, evaluation code followed in March, and an updated dataset was posted on 2024-07-17 after acceptance to ACL 2024. It has no official successor and remains a broad complement to math-only sets such as Omni-MATH.

Further reading