SCROLLS

Overview

SCROLLS is a standardized long-sequence benchmark suite. It packages seven naturally long English NLP tasks into a unified text-to-text format: GovReport, SummScreenFD, QMSum, NarrativeQA, Qasper, QuALITY, and ContractNLI. The suite was created because many earlier NLP benchmarks used sentence- or paragraph-length inputs, while real applications often require reading reports, scripts, contracts, papers, or meetings.

Task details

The tasks span summarization, question answering, multiple-choice reading comprehension, and natural language inference. Inputs are long documents or document-like contexts; outputs are task-specific targets such as summaries, answers, labels, or answer options.

Example

For qmsum, the model receives a meeting transcript and a query asking for a summary of a topic or participant’s position. For contract_nli, the model reads a non-disclosure agreement and judges whether a legal hypothesis follows from it. SCROLLS standardizes both into the same input / output form.

Scoring methodology

SCROLLS uses task-specific metrics and aggregates them for leaderboard comparison. That aggregate is useful for broad tracking but hides important behavior: a model can improve summarization and regress on QA, or vice versa. Per-task reporting is mandatory for serious comparisons.

Paper Table 2 baseline stats

The visual graph on this page shows only the Avg statistic from Table 2 of the official SCROLLS research paper. The paper defines this final SCROLLS score by averaging each dataset’s overall performance score. The graph does not show per-dataset results for GovReport, SummScreenFD, QMSum, NarrativeQA, Qasper, QuALITY, or ContractNLI, and it does not mix in later leaderboard submissions.

Source: SCROLLS paper, Table 2: SCROLLS: Standardized CompaRison Over Long Language Sequences.

What it doesn’t measure

SCROLLS does not isolate retrieval depth, positional robustness, or million-token scaling. It evaluates naturally long text understanding around the component datasets it wraps.

Known issues & criticisms

• The aggregate score mixes unlike metrics.
• Several component datasets are themselves public and static.
• Some tasks reward lexical overlap more than factual consistency.

Contamination notes

Risk is high for modern frontier claims because the suite and its component datasets are public. SCROLLS remains valuable for regression testing and cross-task diagnostics, but not as a private contamination-resistant leaderboard.

How to run it

from datasets import load_dataset

dataset = load_dataset("tau/scrolls", "qasper")
# Options include:
# gov_report, summ_screen_fd, qmsum, narrative_qa, qasper, quality, contract_nli

The official repository also includes evaluator and baseline folders:

git clone https://github.com/tau-nlp/scrolls.git
cd scrolls

History & lineage

SCROLLS was submitted in January 2022 and published at EMNLP 2022. ZeroSCROLLS later adapted the idea for zero-shot LLM evaluation with no training data and several new tasks.

Further reading

• SCROLLS paper: https://arxiv.org/abs/2201.03533
• Official repository: https://github.com/tau-nlp/scrolls
• Hugging Face dataset card: https://huggingface.co/datasets/tau/scrolls
• Official website and leaderboard: https://www.scrolls-benchmark.com/