benchwiki
long-context

QuALITY

Nearing saturation

Multiple-choice long-document QA where writers and validators read the full article, with a hard subset designed to defeat skimming.

last updated 2026-08-12

Performance over time

No performance time series — this benchmark has no cleanly-sourced, metric-consistent aggregate scores to plot. See the prose below for why, and the facts sidebar for what it measures.

Overview

QuALITY is a multiple-choice benchmark for long-document comprehension. Its contexts are English articles and stories averaging about 5,000 tokens. Questions are written and validated by contributors who read the full passage, and the dataset includes a “hard” flag for questions that annotators usually missed when forced to skim.

The benchmark is valuable because it explicitly tests whether a model can read enough of the document, not just search for a short lexical match.

Task details

Each item has a long article, a question, four answer options, and a gold label. The official data also records validation annotations and whether the question belongs to the hard subset. A model chooses one answer option.

Example

A representative QuALITY item asks about an event or inference in a long story. The distractors are designed to look plausible unless the reader understands the relevant passage. A model that only retrieves a keyword may choose the wrong option when several characters or events share similar wording.

Scoring methodology

The primary metric is exact multiple-choice accuracy. The hard subset should be reported separately because it captures the intended anti-skimming pressure. QuALITY is easier to score than open-ended QA, but it can still be sensitive to prompt format, option order, and whether the model is asked to reason before choosing.

What it doesn’t measure

QuALITY does not test answer generation, citation grounding, or million-token context handling. Its average context length was long for 2021; it is not enough by itself to evaluate modern million-token claims.

Known issues & criticisms

  • Static public training/dev data creates leakage risk.
  • Multiple-choice format can reward elimination strategies rather than full comprehension.
  • The hard subset is based on human speed validation, not a formal reasoning taxonomy.

Contamination notes

Risk is medium. Test annotations are withheld for the official leaderboard, but the benchmark is static and the public repository exposes enough data and article sources for training contamination to matter.

How to run it

git clone https://github.com/nyu-mll/quality.git
cd quality

For the SCROLLS wrapper:

from datasets import load_dataset

dataset = load_dataset("tau/scrolls", "quality")

History & lineage

QuALITY appeared in 2021 and was published at NAACL 2022. It later became a component task in SCROLLS, ZeroSCROLLS, and LongBench-style long-context evaluations.

Further reading