MMLU

Overview

MMLU (Massive Multitask Language Understanding) is the benchmark that defined “how smart is this model?” for the first half of the 2020s. Introduced by Dan Hendrycks and collaborators in September 2020, it tests breadth of world knowledge and problem-solving through 14,042 multiple-choice questions across 57 subjects — from elementary mathematics and US history to professional law, college medicine, and moral scenarios. Questions were collected from freely available sources online: practice exams for the GRE, USMLE, AP tests, and university course materials.

Its historical importance is hard to overstate. When it launched, GPT-3 — the most capable model in the world — scored 43.9%, barely above the 25% random-guess floor on many subjects, while the authors estimated expert-level human performance at roughly 90%. That gap made MMLU the de facto scoreboard for the scaling era: Gopher, Chinchilla, PaLM, GPT-4, and nearly every frontier release between 2021 and 2024 led its announcement with an MMLU number. The 5-shot MMLU score became shorthand for general capability in a way no single benchmark has managed since.

By mid-2024 the race was over. Frontier models converged in the 86–92% band, colliding with the benchmark’s own error rate: audits found that a meaningful fraction of questions have wrong or ambiguous gold answers, meaning the effective ceiling sits near 90% and differences between top models are statistically meaningless. MMLU remains useful as a sanity check for small and mid-sized models, and as the shared ancestor of an entire family of successors — but it no longer differentiates the frontier.

Task details

Each question presents a stem and exactly four answer options (A–D), with one correct answer. The model’s job is to output the letter of the correct option.

Input: subject-specific instruction (“The following are multiple choice questions (with answers) about {subject}.”), optionally five in-domain example Q&A pairs (the dev split), then the test question with its four options.
Output: a single letter, A–D. Two extraction conventions exist: generative (parse the letter from free text) and log-likelihood (compare the probability of each option letter as the next token). The two can disagree by 1–3 points for the same model — a classic source of leaderboard discrepancies.
Subjects: 57, grouped in the paper into four categories — STEM, humanities, social sciences, and “other” (business, health, misc.). Difficulty spans elementary to professional level.
Splits: dev (5 questions/subject, used as few-shot examples), validation (1,531), test (14,042).

Example

From the college physics subject:

Question: The muon decays with a characteristic lifetime of about 10⁻⁶ second into an electron, a muon neutrino, and an electron antineutrino. The muon is forbidden from decaying into an electron and just a single neutrino by the law of conservation of

(A) charge (B) mass (C) energy and momentum (D) lepton number

Gold answer: D

A frontier model answers this correctly and instantly. In 2020, GPT-3 performed near chance on this subject — the question genuinely separated models. Today the informative failures are almost entirely in questions with disputed gold labels (see Known issues), not in genuine knowledge gaps.

Scoring methodology

Accuracy via exact match on the predicted option letter — no partial credit, no judge model. The subtleties that actually cause score disagreements between reports:

Macro vs micro averaging. The paper macro-averages across the 57 subjects; several labs micro-average across all 14,042 questions. Because subjects have very different sizes (100–1,534 questions), the two can differ by up to a point.
Answer extraction. Log-likelihood scoring vs generated-text parsing (see Task details) produces systematically different numbers, especially for chat-tuned models that preface answers with reasoning.
Prompt format sensitivity. Scores swing 1–4 points based on option formatting, whether the phrase “Answer:” is present, and few-shot example ordering. This is why “MMLU 88.7” from two different harnesses are not the same measurement.
CoT variants. Some 2023–2024 reports quote chain-of-thought or maj@32 MMLU numbers, which are not comparable to the standard 5-shot direct protocol. Gemini Ultra’s headline 90.0% used CoT@32; its 5-shot score was lower. Always check the footnote.

What it doesn’t measure

MMLU measures recognition of knowledge in a four-option format — not reasoning depth, not generation quality, not calibration, and not knowledge retrieval in open-ended settings. A model can score 88% on MMLU and still hallucinate biographical facts freely, because choosing among four options is a fundamentally easier task than producing a fact unprompted (that gap is what SimpleQA targets). It is also English-only, exam-style, and Western-curriculum-centric; strong MMLU performance says little about multilingual or culturally-localized knowledge. Finally, the four-choice format means a well-calibrated eliminator can score ~50% knowing only how to rule out two distractors.

Known issues & criticisms

Label errors. The MMLU-Redux audit (Gema et al., 2024, arXiv:2406.04127) manually re-annotated a 3,000-question subset and found roughly 6.5% of questions contain errors — wrong gold answers, no correct option, multiple correct options, or unparseable stems. The virology subject is notoriously bad, with error estimates above 25%. This puts a hard ceiling on meaningful scores and explains why no model ever “solves” MMLU.
Contamination. The full test set has been public since 2020 and is embedded across the open web (quiz sites, GitHub mirrors, forum discussions). Decontamination by n-gram matching demonstrably misses paraphrased leakage. Post-2022 model scores should be read as an upper bound on true knowledge.
Prompt-format gaming. Because scores are sensitive to formatting, vendors can (and do) select favorable evaluation configurations. The absence of a single canonical harness makes cross-vendor comparison unreliable at the ±2pt level.
Answer-choice bias. Models exhibit position bias (over-selecting certain letters), and shuffling options changes scores — evidence that some performance reflects format priors rather than knowledge.
Shortcut solvability. Studies showed models score well above chance on some subjects when shown only the answer options without the question, indicating distractor-quality problems.

Contamination notes

Risk: high, and unmitigable. The test set is static, fully public, MIT-licensed, and has circulated for six years. It appears verbatim and paraphrased in Common Crawl-derived corpora. Unlike refresh-cycle benchmarks (LiveCodeBench, yearly AIME), MMLU has no uncontaminated variant of itself — the family’s answer to contamination was to build harder successors (MMLU-Pro) rather than refresh the original. Treat any modern MMLU score as memorization-inclusive.

How to run it

lm-evaluation-harness (the closest thing to a community standard):

pip install lm-eval
lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-3.1-8B \
  --tasks mmlu \
  --num_fewshot 5 \
  --batch_size auto

Subject-level tasks are available as mmlu_abstract_algebra, mmlu_anatomy, etc.; category groups as mmlu_stem, mmlu_humanities, mmlu_social_sciences, mmlu_other.

Direct from HuggingFace:

from datasets import load_dataset
ds = load_dataset("cais/mmlu", "all")   # or a single subject, e.g. "college_physics"
print(ds["test"][0])
# {'question': ..., 'choices': [...], 'answer': 3, 'subject': ...}

Report: harness name, shot count, extraction method (loglikelihood vs generative), and averaging mode alongside any score.

History & lineage

2020-09 — Paper released; GPT-3 at 43.9% vs ~90% expert estimate frames the decade’s capability gap.
2021–2022 — Scaling-law era scoreboard: Gopher (60.0), Chinchilla (67.6), PaLM (69.3). Chinchilla beating the 4× larger Gopher on MMLU was key evidence for compute-optimal scaling.
2023-03 — GPT-4 hits 86.4%, within striking distance of the expert baseline; MMLU begins its transition from “frontier gap” to “table stakes.”
2024 — Frontier convergence at 86–92%; MMLU-Redux documents the label-noise ceiling. MMLU-Pro (10 options, reasoning-heavier, cleaner labels) launches as the designated successor. Global-MMLU and MMMLU extend it multilingually.
2025→ — Frontier labs largely stop reporting MMLU in launch posts. The knowledge-benchmark frontier moves to GPQA Diamond and Humanity’s Last Exam.

If you’re choosing a knowledge benchmark today: use MMLU-Pro for mid-tier model comparison, GPQA Diamond or HLE for the frontier, and keep MMLU as a cheap regression test for small models.

shots	5-shot
reasoning	direct
temperature	0
pass@k	pass@1
tools	no
harnesses	lm-evaluation-harness `mmlu` HELM `mmlu` Inspect `mmlu`

Performance over time