benchwiki
multimodal

Video-MME

Deprecated
2024-05-31 → today

Video-understanding benchmark with 900 videos, 2,700 questions, three duration bands, subtitles, and audio.

last updated 2026-09-05

Performance over time

vendor-reported independent
7080902024

Baseline: No single protocol-comparable human aggregate for the original Video-MME is recorded here.

Overview

Video-MME broadened multimodal evaluation from still images to sequential visual data. It contains 900 videos totaling about 254 hours and 2,700 human-written multiple-choice questions. Videos span six domains, 30 subfields, and three duration bands: short (under two minutes), medium (four to fifteen minutes), and long (thirty to sixty minutes).

The dataset also provides subtitles for 744 videos and audio for all 900, enabling controlled comparisons between frames-only and richer multimodal inputs. It was influential, but its maintainers now position Video-MME-v2 as the next-stage benchmark.

Task details

Each video has three four-option questions covering 12 task types across perception, reasoning, and information synthesis. Models sample frames or ingest video directly, optionally receive subtitles or native audio, and return one option letter.

The duration bands are balanced at 300 videos and 900 questions each. Videos range from 11 seconds to one hour.

Example

A long-video question may ask which event occurred after another event shown much earlier. Answering requires locating both moments and preserving their temporal order; a single uniformly sampled frame can miss either clue. The official paper also studies whether subtitles or audio recover information absent from sampled frames.

Scoring methodology

The headline condition for this page is with subtitles — multiple-choice accuracy with subtitle text supplied — which is the leaderboard’s default sort at https://video-mme.github.io/home_page.html#leaderboard. Results are reported overall and by short, medium, and long duration, with w/o subs and w subs shown side by side.

The metric is simple; the input pipeline is not. Frame count, resolution, sampling rate, maximum-frame cap, subtitle formatting, and native audio support can all change the effective information available. For example, the same GPT-4o run scores 71.9% without subtitles and 77.2% with subtitles (384 frames), and Gemini 1.5 Pro scores 75.0% without and 81.3% with (1/0.5 fps), so the subtitle condition must accompany every score.

What it doesn’t measure

Video-MME does not require open-ended generation, dense temporal localization, consistent answers to linked questions, or interactive navigation through a video. Many systems evaluate sampled frames rather than native continuous video, so the score can reflect the sampler as much as the model.

Known issues & criticisms

  • Public YouTube videos can disappear or change; the maintainers have had to replace broken links and re-annotate items.
  • Different frame budgets make leaderboard values non-comparable even when both are labeled Video-MME.
  • Subtitles can turn audiovisual understanding into long-text retrieval, while frames-only runs omit information available in speech.
  • The successor repository states that existing benchmark scores were saturating and could diverge from actual user experience. Video-MME-v2 addresses this with linked question groups and nonlinear consistency/coherence scoring.

Contamination notes

The questions were newly annotated and filtered so Gemini 1.5 Pro scored below 15% from question text alone during construction. That is useful evidence against simple textual priors, but it does not prevent post-release contamination. The videos and annotations are public and static, so present-day exposure risk is high.

How to run it

git clone https://github.com/MME-Benchmarks/Video-MME.git
cd Video-MME

Follow the repository’s current data-download and evaluation instructions. Record the exact video source revision, frame sampler, frame count and resolution, subtitle/audio condition, and per-duration results. For new evaluations, use Video-MME-v2 unless compatibility with historical claims is required.

History & lineage

Video-MME was submitted on 2024-05-31 and later accepted at CVPR 2025. A June 2024 refresh replaced fragile links and updated model frame sampling. In April 2026 the team launched Video-MME-v2, explicitly citing saturation and shortcomings in existing evaluation paradigms. The v2 score is not comparable: it uses groups of four related questions and nonlinear scoring for consistency and reasoning coherence.

Further reading