benchwiki
multimodal

AV-SpeakerBench

Active
2025-12-01 → today

Speaker-centric audiovisual benchmark for aligning who speaks, what is said, and when in real multi-person videos.

last updated 2026-09-05

Performance over time

vendor-reported independent
4555657585952025human baseline 93.74%

Baseline: Paper-reported human overall accuracy under the benchmark protocol.

Overview

AV-SpeakerBench asks a sharper question than broad video QA: can a model bind speech to the visible person who produced it, and reason about that speech in time? It contains 3,212 expert-curated, four-choice questions over 2,051 real-world videos and covers 12 task types centered on visible speakers, visual events, and vocal attributes.

The benchmark is designed so auditory and visual evidence are coupled in the question semantics. Its paper reports a 93.74% human score versus 73.04% for Gemini 2.5 Pro Thinking, leaving substantial headroom.

Task details

Questions test speaker recognition and counting, attributes and activities, visual counting, speech-content recognition, utterance duration, pitch, rate, intensity, and counting. Each task has at least 200 validated examples.

The canonical input is synchronized audio and video. The authors also evaluate visual-only and audio-only ablations to determine whether systems genuinely benefit from fusion.

Example

A question may identify a visible person by clothing, anchor a moment to something that person says, and ask what another person does immediately afterward. The correct choice requires matching voice to identity, locating the utterance, and inspecting the temporally adjacent visual event.

Scoring methodology

AV-SpeakerBench uses exact four-choice accuracy, reported overall and per task. The paper evaluates models through native audiovisual interfaces with identical inputs and prompts where possible. It also varies frame sampling (8 frames, 16 frames, and roughly 1 fps) and reports modality ablations.

The overall score should never stand alone: task-level and ablation results reveal whether a model is hearing speech, tracking visible speakers, or exploiting one modality. The paper found that stronger sampling slightly improved scores while preserving model ordering.

What it doesn’t measure

The suite is about human speech in visible multi-person scenes. It does not broadly test environmental sounds, music understanding, non-speech acoustic events, ASR word error rate, speech generation, multilingual conversation, or interactive dialogue. Multiple-choice accuracy also does not assess calibrated uncertainty or explanation fidelity.

Known issues & criticisms

  • The authors acknowledge that some questions can occasionally be solved from mouth motion, gaze, or gestures without audio. They interpret this as legitimate visual speech understanding, but it complicates claims that every item strictly requires both modalities.
  • Native audiovisual APIs differ in compression, frame sampling, audio handling, and maximum duration, so a nominally identical prompt may not deliver identical evidence.
  • The public static test set will become more vulnerable to contamination over time.
  • The benchmark is English-focused and speaker-centric, so it does not represent the full audiovisual world.

Contamination notes

The construction pipeline starts with real videos, creates fusion-grounded questions, checks temporal precision and speaker identity, and removes trivially solvable items. These controls reduce shortcut risk at launch. Because the final CSVs and modality-specific prompts are public, later training exposure remains possible; retain exact dataset revisions in reports.

How to run it

Load the official dataset and choose the matching modality condition:

from datasets import load_dataset

dataset = load_dataset("plnguyen2908/AV-SpeakerBench", split="test")

The repository provides audiovisual, visual_only, and audio_only prompt/data layouts. Use the canonical audiovisual condition for the headline score and report frame rate or frame cap, reasoning mode, and per-task accuracy.

History & lineage

AV-SpeakerBench was submitted on 2025-12-01. Version 2, posted in 2026, includes additional analyses such as Gemini 3 Pro, frame-sampling comparisons, and examination of questions solvable from visual cues alone. It is complementary to broad video suites such as Video-MME, not a score-compatible successor.

Further reading