Overview
LongBench-Pro is a newer continuation of the LongBench line. It targets a gap left by earlier long-context benchmarks: synthetic tasks scale cheaply but can be unrealistic, while fully manual long-document annotation is expensive. LongBench-Pro uses a human-model collaborative construction pipeline: frontier models draft challenging questions and answers, while experts validate and refine them.
The result is a bilingual English/Chinese benchmark of 1,500 naturally occurring long-context samples across 11 primary tasks and 25 secondary tasks, with input lengths from 8k to 256k tokens.
Task details
Each sample includes a long context, language, token-length bucket, primary and secondary task labels, context requirement, non-thinking and thinking prompts, reference answer components, and difficulty. The benchmark labels context requirement as full-context or partial-context dependency and stratifies length into six levels and difficulty into four levels.
Example
A partial-context task may ask for a localized answer from a long document. A full-context task may require integrating evidence across distant spans, such as comparing structured information, detecting anomalies, or reasoning over a long report.
Scoring methodology
LongBench-Pro uses task-specific metrics rather than one universal grader. Retrieval and ranking, sequencing or clustering, QA, citation, violation detection, generation, and summarization-style tasks each require their own scoring rule, normalized into the aggregate. The official repository supports both thinking and non-thinking prompt variants.
Paper Overall leaderboard stats
The visual graph on this page shows only the Overall statistic from the supplied official LongBench-Pro paper table. Reasoning / thinking status is shown in the table for interpretation, but the graph itself plots only the numeric Overall score. It does not show per-task, per-language, per-length, difficulty-sliced, or thinking-vs-non-thinking prompt breakdowns.
| Rank | Model | Type | Reasoning / Thinking | Overall shown in graph |
|---|---|---|---|---|
| 1 | Gemini-2.5-Pro | Thinking | YES - Explicit Thinking | 73.42 |
| 2 | GPT-5 | Thinking | YES - Explicit Thinking | 72.61 |
| 3 | Qwen3-235B-A22B-Thinking-2507 | Thinking | YES - Explicit Thinking | 66.97 |
| 4 | Qwen3-Next-80B-A3B-Thinking | Thinking | YES - Explicit Thinking | 63.95 |
| 5 | DeepSeek-R1-0528 | Thinking | YES - Explicit Thinking | 61.89 |
| 6 | DeepSeek-R1 | Thinking | YES - Explicit Thinking | 60.07 |
| 7 | Qwen3-30B-A3B-Thinking-2507 | Thinking | YES - Explicit Thinking | 59.68 |
| 8 | Claude-4-Sonnet | Mixed | Mixed - Not explicitly Thinking | 56.07 |
| 9 | Gemini-2.5-Flash | Mixed | Mixed - Not explicitly Thinking | 55.92 |
| 10 | MiniMax-M2 | Thinking | YES - Explicit Thinking | 53.21 |
In this top-10 slice, 8 models are explicitly labeled Thinking and 2 are Mixed / non-explicit. The top 7 ranked models are all explicitly labeled Thinking.
Source: LongBench-Pro paper: LongBench Pro: A More Realistic and Comprehensive Bilingual Long-Context Evaluation Benchmark.
What it doesn’t measure
LongBench-Pro does not eliminate contamination risk because the data is public. It also does not reduce all long-context ability to one number; the intended use is sliced analysis by length, task, language, difficulty, and context requirement.
Known issues & criticisms
- Public natural documents can leak into training.
- Model-generated draft construction requires expert validation to avoid hidden artifacts.
- The aggregate score is less informative than the benchmark’s multidimensional breakdowns.
Contamination notes
Risk is high for the released public split. For high-stakes model claims, require disclosure of training-data filtering and inspect performance by task and length, not just aggregate score.
How to run it
git clone https://github.com/caskcsg/longcontext.git
cd longcontext/LongBench-Pro
python3 -m venv longbench_pro
source longbench_pro/bin/activate
pip install -r requirements.txtLoad the dataset:
from datasets import load_dataset
dataset = load_dataset("caskcsg/LongBench-Pro", split="test")History & lineage
LongBench-Pro was submitted in January 2026. It follows LongBench v1 and LongBench v2, keeping the broad long-context mission but emphasizing natural documents, bilingual coverage, 8k-256k length stratification, and expert-validated human-model collaborative construction.
Further reading
- LongBench-Pro paper: https://arxiv.org/abs/2601.02872
- Official repository: https://github.com/caskcsg/longcontext/tree/main/LongBench-Pro
- Hugging Face dataset: https://huggingface.co/datasets/caskcsg/LongBench-Pro
- Original LongBench paper: https://arxiv.org/abs/2308.14508