QMSum

Overview

QMSum turns meeting summarization into a query-focused long-context task. Instead of asking for one generic summary of a meeting, it asks for a summary targeted to a user query, such as the whole meeting, a topic, a decision, or a participant’s view. The benchmark covers 232 meetings and 1,808 query-summary pairs across academic, product-design, and government committee domains.

This matters because meetings are long, multi-speaker, and topic-shifting. A model has to identify relevant turns before it can summarize them, which makes QMSum a useful bridge between retrieval and generation.

Task details

Each example includes a meeting transcript, topic metadata, and a query. The model produces a concise answer summary grounded in the relevant meeting span. Specific-query examples include annotated relevant spans; whole-meeting queries require broader synthesis.

Example

In a committee transcript, a query may ask what a named participant thought about a policy discussion. The model must find the relevant turns from that speaker and summarize the position. A generic meeting summary that omits the queried person fails the point of the task even if it is fluent.

Scoring methodology

QMSum is normally scored with ROUGE-style overlap against human reference summaries. This gives an inexpensive automatic signal, but it rewards lexical overlap and can under-credit valid paraphrases. For model comparisons, report the exact split, prompt, truncation policy, and metric version.

Paper Table 3 R-1 stats

The visual graph on this page shows only ROUGE-1 / R-1 from Table 3 of the official QMSum paper and repository. It does not show R-2 or R-L.

* indicates the model uses spans extracted by the paper’s Locator as input. (gold spans) indicates the summarizer uses gold spans as input.

Source: QMSum paper, Table 3: QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization, and the official Yale-LILY/QMSum repository.

What it doesn’t measure

QMSum does not measure real-time meeting assistance, diarization from audio, or factual consistency beyond overlap with the reference summary. It assumes transcripts are already available.

Known issues & criticisms

• ROUGE is a weak proxy for summary usefulness and factual faithfulness.
• Long transcripts make truncation strategy a major experimental variable.
• The static public split is vulnerable to post-release contamination.

Contamination notes

Risk is high for frontier-model claims because the official data is public and reused inside multiple long-context suites. Treat it as a diagnostic summarization task, not as a private held-out benchmark.

How to run it

git clone https://github.com/Yale-LILY/QMSum.git
cd QMSum

The official repository exposes data/ALL plus domain-specific folders. In SCROLLS:

from datasets import load_dataset

dataset = load_dataset("tau/scrolls", "qmsum")

History & lineage

QMSum was introduced at NAACL 2021. SCROLLS later wrapped it as one of seven long-sequence tasks; ZeroSCROLLS adapted it for zero-shot long-text evaluation; LongBench includes it as a summarization component.

Further reading

• Original QMSum paper: https://arxiv.org/abs/2104.05938
• Official QMSum repository: https://github.com/Yale-LILY/QMSum
• SCROLLS dataset card: https://huggingface.co/datasets/tau/scrolls
• ZeroSCROLLS dataset card: https://huggingface.co/datasets/tau/zero_scrolls