benchwiki
knowledge

MMLU-Redux

Saturated

A manual error audit of MMLU: experts re-annotated thousands of questions and found ~6.5% are broken — the evidence base for MMLU's label-noise ceiling.

last updated 2026-07-21

Performance over time

No performance time series — this benchmark has no cleanly-sourced, metric-consistent aggregate scores to plot. See the prose below for why, and the facts sidebar for what it measures.

Overview

MMLU-Redux is not a benchmark models compete on — it is a benchmark’s autopsy. In June 2024, Gema et al. at Edinburgh asked the question in their title, “Are We Done with MMLU?”, and answered it not with a new test but with a manual error audit of the old one. Human annotators re-examined MMLU questions one by one, classifying each with a formal taxonomy of failure modes: wrong gold answer, no correct option, multiple correct options, unclear question, unclear options. The original audit covered 3,000 questions across 30 subjects with 14 expert annotators; the expanded Redux 2.0 now spans 5,700 questions across all 57 MMLU subjects.

The verdict was damning and precise: roughly 6.5% of MMLU questions contain errors — and the damage is wildly concentrated. Some subjects are nearly clean; others are riddled. Virology is the notorious worst case, with error rates the original 30-subject audit put above 25% and the full Redux 2.0 audit measured as high as ~57%. Because a benchmark’s effective ceiling is set by its broken questions — no model can score above the fraction that are answerable — this is the formal evidence base for a claim repeated across this wiki: label noise, not model capability, caps MMLU near 90%. When the MMLU page says top-model gaps are “within noise,” MMLU-Redux is the measurement that noise refers to.

That makes MMLU-Redux a genuinely different kind of entry — benchmark forensics. Its status here is saturated, and the reasoning is worth stating plainly because it sets precedent. As a competition benchmark, Redux is just error-corrected MMLU: it inherits MMLU’s contamination and frontier convergence, so it cannot differentiate strong models — saturated, by this wiki’s definition of status as usefulness-as-a-model-differentiator. But its other role — as an actively-used corrected reference set and a reusable audit methodology — is very much alive, and that role is documented throughout this page. The status badge describes the eval; the prose describes the tool. (The tension is real, and future audit or contamination-detection entries should resolve it the same way: status tracks discriminating power, prose carries the living methodology.)

Task details

MMLU-Redux is the MMLU question set with corrected annotations, so the task a model faces is identical to MMLU: read a question, pick the correct option among four, in the standard direct-answer protocol. What differs is the annotation layer — every included question has been human-verified, and errors are labelled with a taxonomy rather than silently accepted.

The error taxonomy has two groups:

  • Question assessment — Bad Question Clarity (poorly worded, grammatically broken, or missing information) and Bad Options Clarity (options unclear, redundant, or irrelevant).
  • Ground-truth verification — No Correct Answer (none of the options is right), Multiple Correct Answers (more than one option is defensible), and Wrong Ground Truth (MMLU’s labelled answer is simply incorrect).

dataset.size is 5,700 — the Redux 2.0 configuration covering all 57 subjects, which is the current primary reference — following the wiki’s size-matches-primary-configuration convention (see GPQA). The original 3,000-question / 30-subject audit is the figure cited on the MMLU page and remains the historical baseline for the ~6.5% and virology->25% numbers.

Example

A Wrong Ground Truth case — the most consequential error type — looks like an ordinary question with a bad answer key:

Question (Virology, illustrative): Which of the following is the primary transmission route of the pathogen described?

MMLU gold answer: an option that domain experts judge incorrect. Redux annotation: Wrong Ground Truth — the labelled-correct option contradicts the field’s established consensus; a different option is correct.

For such a question, a model that “knows” the right answer is marked wrong by MMLU’s key, and a model that parrots the mislabeled answer is marked right. Aggregate this over 6.5% of questions — over 25% in the worst subjects — and you see why the top of the MMLU leaderboard is noise: models are being graded partly on their agreement with mistakes.

Scoring methodology

The primary metric is ordinary accuracy on the corrected question set — but the paper’s real contribution is a comparative analysis, and that is what to read Redux for:

  • Reported vs corrected accuracy. Re-running models against the corrected labels changes their measured scores, and — more importantly — changes their relative ranking. The paper’s headline illustration: on the Virology subset, Palmyra X v3 ranked 4th when scored on all instances but 1st when scored only on the correct (non-broken) instances. Rankings built on noisy labels can invert once the noise is removed.
  • Correct-instances-only evaluation. A robust way to use Redux is to score models only on questions that passed audit, removing the broken items rather than trusting their keys. This is the “clean subset” use that keeps Redux actively useful even though the aggregate is saturated.
  • Accuracy is still MC exact-match, in MMLU’s direct-answer protocol — Redux deliberately does not change the task, only the labels, so its numbers stay comparable to MMLU’s (unlike MMLU-Pro, which changes the protocol on purpose).

What it doesn’t measure

MMLU-Redux does not introduce new capabilities to measure — it measures the measurement. It is silent on reasoning, generation, tools, or anything MMLU itself doesn’t cover; it adds no new domains and no harder questions (that is MMLU-Pro’s job). It is not a frontier discriminator and should never be used to rank strong models by their aggregate Redux score — its entire value is diagnostic: quantifying how much of a benchmark’s apparent signal is actually label noise, and providing a cleaned reference for fairer evaluation. It is English-only, and it inherits every contamination problem of its parent.

Known issues & criticisms

  • Not a competition benchmark. Read as a leaderboard, Redux is just saturated MMLU; its aggregate accuracy differentiates nothing at the frontier. Using it as a scoreboard misunderstands it.
  • Annotation is itself a judgment call. Deciding that a gold answer is “wrong” or a question is “unclear” requires expert judgment, and some Redux annotations are themselves contestable — audits can disagree, so the ~6.5% figure has its own (smaller) uncertainty.
  • Inherits MMLU’s contamination fully. These are the same public questions, so any memorization affecting MMLU affects Redux; correcting labels does not decontaminate.
  • Coverage grew but is still a sample. Even Redux 2.0’s 5,700 questions are a subset of MMLU’s ~14,000; per-subject error rates are estimates, tighter for large subjects than small ones.
  • Fixing labels doesn’t fix saturation. A clean MMLU is still an easy, recognition-based test for frontier models; Redux raises the validity of MMLU scores without raising the ceiling.

Contamination notes

Risk: high, entirely inherited. MMLU-Redux is MMLU’s questions with better annotations, so everything true of MMLU’s contamination is true here: public since 2020, mirrored across the web, fully static, ungated. Re-annotating the answer keys changes what “correct” means but does nothing to reduce how thoroughly the questions sit in training corpora — and Redux’s own prominence has only increased how much these specific items are discussed online. The value of Redux is not contamination resistance (it has none) but contamination accounting: it lets you separate “the model memorized the answer” from “the answer was wrong anyway.” For a contamination-resistant knowledge test, use a newer, gated, or refreshed benchmark — Redux is a lens on the old one, not a replacement for it.

How to run it

MMLU-Redux is a dataset, not a bespoke harness — you evaluate it with the standard MMLU protocol on the corrected questions:

from datasets import load_dataset
# Redux 2.0: 5,700 re-annotated questions across all 57 subjects
ds = load_dataset("edinburgh-dawg/mmlu-redux-2.0")
# each item carries the original question, options, MMLU gold answer,
# and the Redux error-type annotation (e.g. "wrong_groundtruth", "ok")

The most useful pattern is correct-instances-only scoring: filter to questions annotated ok (passed audit), then run your standard MMLU evaluation on that clean subset — giving a label-noise-free accuracy that is fairer than raw MMLU. Report which annotation version (original 3,000 / Redux 2.0 5,700) and whether you filtered or re-scored.

History & lineage

  • 2020 — MMLU launches and becomes the knowledge benchmark of the era.
  • 2023–2024 — Frontier models converge in MMLU’s high-80s band; practitioners suspect the plateau is partly artifactual but lack a formal measurement.
  • 2024-06 — Two papers land days apart and jointly close the MMLU story. This audit, “Are We Done with MMLU?” (Gema et al., arXiv:2406.04127), quantifies the label-noise ceiling — ~6.5% overall, virology worst — providing the evidence that MMLU’s frontier is noise. Its sibling, MMLU-Pro (arXiv:2406.01574), fixes the flaws with harder, cleaner questions. Redux is later published at NAACL 2025 and expanded to Redux 2.0 (all 57 subjects).
  • 2024–2026 — Redux settles into its lasting role: not a leaderboard, but the standard reference for “how much of this benchmark is broken,” and a template for auditing other datasets.

The three MMLU pages tell one story, each standing alone: the original (MMLU), the audit that explained its ceiling (this page), and the successor that fixed its flaws (MMLU-Pro).

Further reading