benchwiki
long-context

QASPER

Nearing saturation

Scientific-paper QA over full NLP papers, with abstractive, extractive, yes/no, and unanswerable answers plus supporting evidence.

last updated 2026-08-12

Performance over time

No performance time series — this benchmark has no cleanly-sourced, metric-consistent aggregate scores to plot. See the prose below for why, and the facts sidebar for what it measures.

Overview

QASPER is a long-document question-answering benchmark built around full NLP research papers. A question writer sees only a paper’s title and abstract, writes an information-seeking question, and a separate annotator answers it from the full paper while marking supporting evidence. That design makes the task closer to research-paper reading than to short-passage QA: the answer can be an extractive span, a free-form explanation, a yes/no answer, or “unanswerable.”

Its long-context value is not simply length. QASPER asks whether a model can use a paper as a source, find relevant sections, and connect answer text to evidence. That is why it appears as a component task in broader suites such as SCROLLS and LongBench.

Task details

Each example contains a paper title, abstract, full text split into sections, figures/tables, and one or more questions. The model must answer the question from the full paper and, when the protocol asks for it, identify evidence paragraphs or figure/table references. The released Hugging Face dataset contains 5,049 questions over 1,585 NLP papers.

Example

A typical prompt asks a question such as “What data is the language model pretrained on?” for a specific NLP paper. A good answer must locate the relevant methodological section and return the paper-grounded answer, not a generic guess about common pretraining corpora. If the paper does not contain the requested fact, the correct behavior is to mark the question unanswerable.

Scoring methodology

QASPER is usually evaluated with answer F1 and evidence F1. Answer scoring is awkward because answer types differ: a yes/no answer, an extractive span, and a free-form explanation are not the same object. Evidence scoring checks whether the model selected the support used by annotators. Report answer and evidence scores separately; a model can produce a plausible answer while failing to ground it in the cited paper.

What it doesn’t measure

QASPER does not measure broad scientific correctness, peer-review quality, or whether the model understands a whole research area. It measures paper-grounded retrieval and answer synthesis under a fixed annotation scheme.

Known issues & criticisms

  • The public static test set is easy to contaminate after release.
  • Scientific papers vary widely in length, structure, and PDF extraction quality, so token budgets and preprocessing choices can change comparability.
  • Evidence F1 rewards matching annotated support, not necessarily every valid supporting passage.

Contamination notes

Risk is high because the dataset is public, static, and widely mirrored through Hugging Face and benchmark suites. For modern model comparisons, QASPER is best treated as a diagnostic subtask rather than a leakage-resistant leaderboard.

How to run it

pip install datasets
from datasets import load_dataset

dataset = load_dataset("allenai/qasper")

For SCROLLS-style evaluation:

from datasets import load_dataset

dataset = load_dataset("tau/scrolls", "qasper")

History & lineage

QASPER appeared in 2021 as a scientific-document QA benchmark. SCROLLS later standardized it into a text-to-text long-sequence format, and LongBench included Qasper as one of its single-document QA tasks.

Further reading