benchwiki
multimodal

MMMU

Nearing saturation
2023-11-27 → today

College-level multimodal questions across 30 subjects and 183 subfields, with interleaved text and diverse images.

last updated 2026-09-05

Performance over time

vendor-reported independent
506070809020232024human baseline 88.6%

Baseline: Best expert validation accuracy. The paper also reports medium and lowest expert groups at 82.6% and 76.2%; no equivalent full-test human score is reported.

Overview

MMMU tests whether multimodal models can solve college-level problems that combine visual perception, specialist knowledge, and deliberate reasoning. Its roughly 11.5K questions cover six broad disciplines, 30 subjects, 183 subfields, and about 30 image types including charts, diagrams, maps, tables, music notation, chemical structures, medical images, photographs, and paintings.

MMMU remains a useful breadth test, but it is no longer the strongest version of its own lineage. The maintainers released MMMU-Pro to remove text-only-solvable items and make the choice format more robust, and in 2026 they published the original test answers.

Task details

An item can contain text interleaved with one or more images. Most questions are multiple choice, while some require a short open response. The six disciplines are Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering.

The suite contains 150 development examples, about 900 validation examples, and 10,500 test examples. The development set provides five demonstrations per subject.

Example

A chemistry item may show a molecular structure or reaction diagram and ask which conclusion follows. Solving it requires reading the visual representation, recalling domain knowledge, and applying it; converting the image to generic caption text may discard decisive details.

Scoring methodology

MMMU uses micro-averaged accuracy. The official evaluator extracts answers with deterministic response-processing rules and regular expressions so that reasoning text does not automatically invalidate an otherwise clear final response. Scores can be broken down by discipline and subject.

Validation and test are different axes and must not be mixed. The timeline on this page uses validation values from the paper because the expert baseline is also validation-only.

What it doesn’t measure

MMMU samples academic test-taking rather than everyday multimodal interaction. It does not evaluate video, audio, embodied action, long-form generation, tool use, or the reliability of explanations. Broad coverage also means relatively few examples per narrow specialty.

Known issues & criticisms

  • Some original items are answerable from text or learned priors without using the image. MMMU-Pro explicitly filters such questions, expands answer choices, and adds a vision-only presentation mode.
  • Source materials come from exams, quizzes, textbooks, and internet resources, creating copyright and training-overlap concerns.
  • Multiple-choice accuracy rewards guessing and may not reflect calibrated expertise.
  • The “expert AGI” framing should not be read as a general intelligence test; it measures a specific academic, image-grounded slice.

Contamination notes

The original server-held test labels initially reduced direct leakage. On 2026-02-12, the official repository released test answers for local evaluation. This improves reproducibility but changes the contamination posture: modern training pipelines can ingest both inputs and labels. For new frontier claims, prefer MMMU-Pro or another controlled set and disclose training filters.

How to run it

git clone https://github.com/MMMU-Benchmark/MMMU.git
cd MMMU
pip install -r requirements.txt

Follow the current mmmu evaluation directory and load MMMU/MMMU from Hugging Face. Use the development samples only for declared few-shot runs, and report split, prompt, image handling, and answer-extraction settings.

History & lineage

MMMU was submitted on 2023-11-27 and published at CVPR 2024. The paper evolved through v4, adding more model and expert results. MMMU-Pro launched in 2024 as a more robust successor. In 2026, the team released the original test answers, making local scoring possible while ending the test set’s secrecy.

Further reading