benchwiki
multimodal

MME

Nearing saturation

Image benchmark with paired yes/no questions across 10 perception and 4 cognition subtasks.

last updated 2026-09-05

Performance over time

No performance time series — this benchmark has no cleanly-sourced, metric-consistent aggregate scores to plot. See the prose below for why, and the facts sidebar for what it measures.

Overview

MME is an early comprehensive benchmark for image-capable large language models. It deliberately uses a simple answer format to test 14 abilities without relying on an LLM judge: 10 perception subtasks cover object existence, count, position, color, OCR, posters, celebrities, scenes, landmarks, and artwork; four cognition subtasks cover commonsense reasoning, numerical calculation, translation, and code reasoning.

Its historical importance is larger than its present frontier difficulty. MME made broad, quantitative comparison of multimodal chat models easy, but its short binary prompts and public static set are now best used as a diagnostic rather than a sole claim of state-of-the-art multimodal ability.

Task details

Each image is paired with two manually written questions: one has a yes answer and the other a no answer. Both use the suffix “Please answer yes or no.” The pairing is intentional: a model that always favors one label can obtain question-level credit but will fail the stricter image-level measure.

The released evaluation set contains 2,374 image-question rows across the 14 subtasks. Perception and cognition are reported as separate totals.

Example

For an object-existence item, the same photograph may receive one true question such as whether a depicted object is present and one false question asking about an absent object. A valid response is only Yes or No; extra explanation can break strict parsers.

Scoring methodology

MME reports ordinary question accuracy and accuracy+, the percentage of images for which both paired questions are correct. Each subtask score is accuracy + accuracy+, so it ranges from 0 to 200. The 10 perception subtasks sum to 2,000 and the four cognition subtasks sum to 800.

Because the benchmark has two official headline axes, this page does not fabricate a single normalized timeline. Reports should state perception and cognition totals separately and identify the exact dataset release.

What it doesn’t measure

MME does not test open-ended visual description, multi-image reasoning, long video, native audio, grounded localization output, or sustained multimodal interaction. Its cognition tasks remain short and its binary response space cannot reveal whether a correct answer came from robust reasoning or a shortcut.

Known issues & criticisms

  • Binary questions are vulnerable to response bias; accuracy+ reduces but does not eliminate this weakness.
  • Several images originate from public datasets, even though the instructions and answers were newly authored.
  • The small per-subtask sample sizes make narrow rank differences unstable.
  • Newer first-party suites address gaps directly: MME-RealWorld uses much larger, high-resolution real-world tasks, while MME-CoT targets reasoning quality.

Contamination notes

At launch, manually authored instruction-answer pairs reduced direct label leakage from the source image datasets. That protection is temporal, not permanent: the complete benchmark has since been public and can appear in training or evaluation-tuning corpora. Treat present-day MME scores as high contamination risk.

How to run it

Download the release and follow the official evaluation folder:

git clone https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models.git
cd Awesome-Multimodal-Large-Language-Models/Evaluation

Generate one answer per image-question row, preserve the official directory/category names, and run the repository’s MME evaluator. Report both perception and cognition totals.

History & lineage

MME was submitted on 2023-06-23. Later paper revisions expanded evaluated models, and v5 was posted in 2025. The maintainers subsequently introduced MME-RealWorld for harder high-resolution scenarios and MME-CoT for chain-of-thought analysis; those are related successors, not score-compatible revisions of the original 14-task suite.

Further reading