Overview
LongBench is a bilingual multitask benchmark for long-context understanding. It covers English, Chinese, and code tasks across six categories: single-document QA, multi-document QA, summarization, few-shot learning, synthetic tasks, and code completion. The original release contains 21 tasks and 4,750 test examples, with most task average lengths in the 5k-15k range.
LongBench became a common first stop for evaluating whether a model can use more than a short prompt. Its weakness is now the same reason it succeeded: it is broad and public, so it is not a clean final exam for post-2023 long-context models.
Task details
Each example has an input, a long context, one or more answers, a length field, dataset name, language, and sometimes class labels. The suite includes natural long documents, multi-document QA, few-shot classification-style tasks, synthetic retrieval tasks, and repository/code completion.
Example
In a multi-document QA task, the model must read several long passages and answer a question whose evidence may be distributed across documents. In a code task, it must use cross-file context rather than a single function body.
Scoring methodology
LongBench reports task-specific scores and averages them by category and overall. That makes it useful for broad comparison, but the average is not a single natural metric. Summarization, QA, synthetic retrieval, and code completion all have different failure modes.
Paper Table 3 baseline stats
The visual graph on this page shows only the OverAll statistic from Table 3 of the official LongBench research paper. Table 3 reports results on few-shot learning, synthetic, and code tasks; OverAll is computed as the macro-average, meaning the mean of Avg, over major task categories. The graph does not show per-task scores, per-category scores, or later LongBench leaderboard submissions.
| Model | OverAll shown in graph |
|---|---|
| GPT-3.5-Turbo-16k | 44.7 |
| Llama2-7B-chat-4k | 26.8 |
| LongChat-v1.5-7B-32k | 31.6 |
| XGen-7B-8k | 25.0 |
| InternLM-7B-8k | 22.6 |
| ChatGLM2-6B | 25.7 |
| ChatGLM2-6B-32k | 41.4 |
| Vicuna-v1.5-7B-16k | 30.5 |
Source: LongBench paper, Table 3: LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding.
What it doesn’t measure
LongBench does not measure very long 100k-1M token robustness as directly as InfiniteBench, RULER, or OpenAI-MRCR. It also does not fully separate retrieval, reasoning, and generation quality.
Known issues & criticisms
- Public static data creates high contamination risk.
- The aggregate score can hide severe task-level weaknesses.
- Original contexts are modest compared with newer long-context claims.
Contamination notes
Risk is high because the benchmark is public, popular, and included in many training/evaluation pipelines. Use it for regression checks and broad profiling, not as decisive evidence of unseen long-context ability.
How to run it
git clone https://github.com/THUDM/LongBench.git
cd LongBench/LongBench
pip install -r requirements.txtfrom datasets import load_dataset
data = load_dataset("THUDM/LongBench", "qasper", split="test")History & lineage
LongBench was released in August 2023 and published at ACL 2024. LongBench v2 later shifted toward deeper realistic reasoning with 503 multiple-choice problems over much longer contexts, and LongBench Pro extended the line with 1,500 bilingual naturally occurring samples across 11 primary tasks.
Further reading
- LongBench paper: https://arxiv.org/abs/2308.14508
- Official repository: https://github.com/THUDM/LongBench
- Original LongBench README: https://github.com/THUDM/LongBench/tree/main/LongBench
- Hugging Face dataset: https://huggingface.co/datasets/THUDM/LongBench
- LongBench Pro paper: https://arxiv.org/abs/2601.02872