Overview
ZeroSCROLLS is the zero-shot successor to SCROLLS. It keeps the long-text focus but removes training data, keeps only test and small validation sets, adapts six SCROLLS tasks, and adds four new tasks, including information-fusing tasks such as sentiment percentage aggregation and chapter-summary ordering.
The benchmark is aimed at LLMs that should be evaluated directly from prompts rather than fine-tuned on benchmark training splits.
Task details
ZeroSCROLLS contains long-text tasks across summarization, QA, multiple-choice QA, multi-hop QA, and aggregation. The dataset card lists tasks such as GovReport, SummScreenFD, QMSum, SQuALITY, Qasper, NarrativeQA, QuALITY, MuSiQue, SpaceDigest, and BookSumSort.
Example
In SpaceDigest, the model reads 50 hotel reviews without ratings and estimates the percentage of positive reviews. In BookSumSort, it receives shuffled chapter summaries and must recover their original order. These tasks force synthesis across many spans rather than retrieving one sentence.
Scoring methodology
ZeroSCROLLS is composite by design. Different tasks use different metrics: summarization uses ROUGE-like scores, QA uses answer metrics, and sorting or aggregation tasks use specialized metrics. Always report task-level scores alongside any aggregate.
Paper Table 3 leaderboard stats
The visual graph on this page shows only the Avg statistic from Table 3 of the official ZeroSCROLLS research paper. Table 3 is the ZeroSCROLLS leaderboard at the time of writing. The graph does not show per-task scores, validation scores, or later leaderboard submissions.
| Model | Avg shown in graph |
|---|---|
| T0pp | 14.3 |
| Flan-T5 | 29.9 |
| Flan-UL2 | 30.6 |
| DaVinci003 | 33.7 |
| ChatGPT | 34.0 |
| Claude | 39.1 |
| GPT-4 | 41.7 |
Source: ZeroSCROLLS paper, Table 3: ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding.
What it doesn’t measure
ZeroSCROLLS does not measure training-time adaptation because it is explicitly zero-shot. It also does not cover million-token contexts; its value is task diversity and zero-shot protocol, not maximum input length.
Known issues & criticisms
- A composite score can hide which long-context skill improved.
- Validation sets are deliberately small and not suitable for model selection.
- Several component datasets have public sources, so leakage risk is not eliminated.
Contamination notes
Risk is medium. The benchmark design reduces training-set overfitting compared with SCROLLS, but the suite is still static and many source datasets are known public artifacts.
How to run it
from datasets import load_dataset
dataset = load_dataset("tau/zero_scrolls", "qasper")Use the official leaderboard protocol for test-set claims; local validation is for smoke-testing prompts and formatting.
History & lineage
ZeroSCROLLS was submitted in May 2023 and published in Findings of EMNLP 2023. It follows SCROLLS but shifts the evaluation framing from supervised long-sequence modeling to zero-shot LLM evaluation.
Further reading
- ZeroSCROLLS paper: https://arxiv.org/abs/2305.14196
- Hugging Face dataset card: https://huggingface.co/datasets/tau/zero_scrolls
- SCROLLS paper: https://arxiv.org/abs/2201.03533
- SCROLLS repository: https://github.com/tau-nlp/scrolls