benchwiki
knowledge

MMLU-Pro

Nearing saturation
2024-06-03 → today

MMLU rebuilt to fix its flaws: 10 options instead of 4, trivia and label noise filtered out, and reasoning-heavy questions that reward chain-of-thought.

last updated 2026-07-21

Performance over time

vendor-reported independent
5565752024

Baseline: No measured human baseline. MMLU-Pro's difficulty was calibrated by model-side filtering (removing questions strong models found trivial and adding reasoning-heavy ones), not by testing humans — so unlike its parent MMLU (famous for its 89.8% expert baseline framing) and unlike GPQA (built around expert vs non-expert gaps), the successor here has no human reference line. A notable irony worth flagging when reading 'above/below human' claims: there is no human number to compare against.

Overview

MMLU-Pro is the benchmark built to repair MMLU, and it is the cleanest parent→successor story on this wiki: nearly every design choice targets a specific, documented MMLU failure. Released by TIGER-Lab (Wang et al.) in June 2024, once frontier models had converged in MMLU’s high-80s band and the benchmark had stopped discriminating, MMLU-Pro re-poses the same broad “know a lot across many subjects” question in a form that the frontier could no longer trivially ace — dropping accuracy by 16 to 33 points relative to MMLU for the same models.

The repairs map one-to-one onto MMLU’s known problems:

  1. Ten answer options instead of four. MMLU’s 4-option format gave a 25% random-guess floor and rewarded distractor-elimination test-taking. MMLU-Pro expands to 10 options, cutting the guess floor to 10% and blunting elimination strategies — so a score reflects knowing the answer, not narrowing to it.
  2. Trivia and label noise filtered; reasoning added. MMLU is riddled with questions strong models find trivial and with mislabeled or ambiguous items (the MMLU-Redux audit put the error rate near 6.5%). MMLU-Pro screens out the easy and the broken and adds reasoning-heavy questions (many sourced from STEM problem sets), shifting the test from recognition toward working-it-out.
  3. Fourteen categories, not fifty-seven micro-subjects. MMLU’s 57 tiny subjects produced per-subject samples so small their confidence intervals were useless (some ±8pt). MMLU-Pro consolidates into 14 broader disciplines, giving each aggregate enough questions to actually mean something.

The empirical payoff that proves the redesign worked: chain-of-thought prompting helps substantially on MMLU-Pro — often by around 20 points — where it barely moved MMLU. On the parent benchmark, “just recall the fact” made reasoning irrelevant; on MMLU-Pro, models that reason beat models that only recognize. That single contrast is the evidence that the new questions demand reasoning rather than memorized recognition.

MMLU-Pro is now nearing saturation (the wiki’s second use of that status, alongside GPQA). Frontier models sit in the high 80s by 2025–26, and the headroom the redesign bought is being spent — but cleaner labels lift its effective ceiling above MMLU’s, so it still separates the strongest systems where MMLU flatlines. It is the right drop-in when you want MMLU’s breadth without MMLU’s saturation, with the caveat that “without saturation” now comes with an expiry date.

Task details

Each question is a 10-option multiple-choice item across 14 disciplines — mathematics, physics, chemistry, biology, health, law, engineering, economics, psychology, business, philosophy, history, and others. There are 12,032 questions in total.

  • Input: a question and ten answer options (A–J). The standard protocol prepends 5 chain-of-thought exemplars (worked answers) before the target question.
  • Output: a reasoning trace followed by a final selected option, from which the answer is parsed.
  • Aggregate: mean accuracy over all 12,032 questions, also broken out per discipline.

Note the dataset.size convention this wiki follows (established on the GPQA page): the structured size field is 12,032, the single reported configuration here — MMLU-Pro ships as one set rather than nested subsets, so there is no subset ambiguity, but the convention (size = the primary reported configuration) is applied consistently.

Example

A physics item illustrates the recognition→reasoning shift. Where an MMLU question might ask you to identify a formula, an MMLU-Pro question makes you use it, and the 10 options include near-miss numerical distractors:

Question: A 2 kg block slides down a frictionless incline of 30° from rest. What is its speed after sliding 4 m along the incline? (g = 9.8 m/s²)

Options (abbreviated): (A) 4.4 m/s (B) 5.0 m/s (C) 6.3 m/s (correct) (D) 6.9 m/s (E) 8.9 m/s … through (J).

The model must compute the vertical drop (4·sin30° = 2 m), apply energy conservation (v = √(2g·h) = √(2·9.8·2) ≈ 6.26 m/s), and match to option (C) — not merely recognize “kinematics.” With ten options, several plausible near-misses (6.9, 5.0) punish arithmetic slips, and the guess floor is 10%. This is why chain-of-thought moves the needle here and did not on MMLU.

Scoring methodology

Scoring is exact-match accuracy on the parsed final option — but the reporting convention differs from MMLU’s in a way that makes the two non-comparable by design:

  • 5-shot chain-of-thought is the standard MMLU-Pro protocol, whereas MMLU’s canonical protocol is direct-answer (often 5-shot, no reasoning). Because CoT is worth ~20 points on MMLU-Pro, a “direct-answer MMLU-Pro” number and a “CoT MMLU-Pro” number are different measurements — and neither is comparable to an MMLU score. Do not read an MMLU-Pro percentage as an MMLU percentage; the whole point was to make the old number obsolete.
  • Answer parsing from CoT output is heuristic (extract the final letter from a reasoning trace); parsing failures can slightly understate a model that reasoned correctly but formatted its answer oddly. Report the harness and extraction method.
  • Ten-option format lowers the guess floor to 10%, so low-capability models score nearer their true knowledge level than on MMLU, where 25% guessing inflated the floor.

What it doesn’t measure

MMLU-Pro measures broad, multi-discipline knowledge applied with reasoning in a multiple-choice format — it does not measure open-ended generation (the options are given), long-horizon or agentic tasks, tool use, calibration, or the correctness of the reasoning itself (only the final option is scored). It is English-only. And despite fixing MMLU’s label noise and triviality, it is still a closed-book multiple-choice test: strong performance means “recognizes-and-reasons-to the right option among ten,” not “can produce the answer unprompted,” which is what free-response science benchmarks like GPQA isolate. It also has no human baseline (see the facts sidebar), so it cannot tell you where models stand relative to people.

Known issues & criticisms

  • No human baseline. Difficulty was set by model-side filtering, not human testing, so MMLU-Pro cannot support any “beats humans” claim — an ironic gap for the successor to the benchmark that popularized expert-baseline framing.
  • Approaching saturation. With the frontier compressed into the high 80s, top-model gaps under ~2 points are again within noise (see the statistical note) — the headroom the redesign created is largely spent.
  • CoT dependence complicates comparison. Because chain-of-thought is worth ~20 points, scores are highly sensitive to whether (and how) reasoning was elicited and parsed; reasoning-tuned models that always “think” blur the setting further.
  • Source contamination risk. Many reasoning questions come from public STEM problem sets and exams, which are exactly the kind of content that leaks into training corpora; the risk is lower than MMLU’s today only because the benchmark is newer.
  • Still closed-book multiple choice. It inherits the structural limits of the MC format, including residual (if reduced) distractor-elimination effects with ten options.

Contamination notes

Risk: medium, and rising. MMLU-Pro is public and ungated on HuggingFace, fully static, and has no refreshed variant — the same structural exposure that eventually undid MMLU. Two things keep it below “high” for now. First, timing: it launched in mid-2024, so it has had far less time in web crawls and training snapshots than MMLU (public since 2020). Second, its 10-option, reasoning-heavy design makes rote memorization of an answer key less sufficient than on a 4-option recall test. Working against it: many questions are drawn from public STEM problem sets whose solutions circulate online, so parts of the benchmark may be indirectly present in training data from the start. The honest trajectory mirrors MMLU’s — medium today, trending toward high as the set ages — which is part of why its status is nearing saturation.

How to run it

lm-evaluation-harness (5-shot chain-of-thought is the standard configuration):

pip install lm-eval
lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-3.1-8B \
  --tasks mmlu_pro \
  --batch_size auto

Official MMLU-Pro repository / dataset:

git clone https://github.com/TIGER-AI-Lab/MMLU-Pro
# or load directly:
from datasets import load_dataset
ds = load_dataset("TIGER-Lab/MMLU-Pro")   # 12,032 questions, 10 options, 14 categories

Always report shots and whether chain-of-thought was used — because CoT is worth ~20 points here, a bare “MMLU-Pro 82” is not a reproducible measurement, and it is never comparable to an MMLU score.

History & lineage

  • 2020 — MMLU sets the template: 57 subjects, 4 options, direct-answer. It defines the knowledge-benchmark era.
  • 2024-06 — Two MMLU papers land days apart and tell a single story. MMLU-Redux (“Are We Done with MMLU?”, Gema et al., arXiv:2406.04127) audits MMLU and quantifies why it saturated — ~6.5% label errors setting a ceiling near 90%. MMLU-Pro (Wang et al., arXiv:2406.01574) fixes MMLU — 10 options, filtered labels, reasoning-heavy questions, 14 categories — and drops frontier accuracy 16–33 points, restoring headroom. Presented at NeurIPS 2024 (Datasets & Benchmarks, Spotlight).
  • 2024–2026 — MMLU-Pro becomes the default “MMLU replacement” on leaderboards; frontier models climb from the low 70s (GPT-4o at launch) into the high 80s, and the benchmark begins nearing saturation in turn.

The three pages form one arc: the original (MMLU), the audit that explained its ceiling (MMLU-Redux), and the successor that fixed its flaws (this page). For frontier discrimination today, MMLU-Pro is nearing its own limit — reach for GPQA (expert-level science) or Humanity’s Last Exam (broad, contamination-resistant) when MMLU-Pro’s headroom runs out.

Further reading