benchwiki
long-context

InfiniteBench

Nearing saturation
2024-02-21 → today

100k+ token benchmark mixing realistic and synthetic tasks across English, Chinese, code, math, and retrieval.

last updated 2026-08-12

Performance over time

vendor-reported independent
10203040502024

Baseline: No single protocol-comparable human baseline is recorded.

Overview

InfiniteBench, written in the paper title as InfinityBench, pushes long-context evaluation beyond 100k tokens. It mixes realistic tasks, such as long-book QA and script speaker identification, with synthetic retrieval, code, and math tasks that can stretch context length further.

The suite is important because many older long-context benchmarks were around 10k tokens. InfiniteBench asks whether models still work when the context crosses the 100k-token line.

Task details

The official task table lists 12 tasks: English summarization, English QA, English multiple-choice QA, English dialogue speaker identification, Chinese QA, code debugging, code execution simulation, long arithmetic calculation, integer search, pass-key retrieval, number retrieval, and key-value retrieval. Average input lengths range from tens of thousands to more than two million tokens for the Chinese QA task.

Example

In Retrieve.PassKey, a hidden key is inserted into a noisy long context and the model must return it. In Code.Debug, the model reads a long code document and selects which function contains a crashing error.

Scoring methodology

Most tasks are scored by accuracy. English and Chinese QA use ROUGE F1, while English summarization uses ROUGE-Lsum. Because task metrics differ, per-task reporting is more meaningful than an aggregate.

Paper Table 3 average stats

The visual graph on this page shows only the Average row from Table 3 of the official InfiniteBench paper: GPT-4 45.63, YaRN-Mistral 19.96, Kimi-Chat 34.73, and Claude 2 37.06. The graph does not show individual subset rows such as retrieval, English QA, Chinese QA, code, or math, because those rows are different benchmark subsets rather than one shared score axis.

Source: InfiniteBench paper, Table 3: InfinityBench: Extending Long Context Evaluation Beyond 100K Tokens.

What it doesn’t measure

InfiniteBench does not isolate positional robustness as cleanly as RULER, and synthetic tasks do not fully represent natural document work. It also does not provide a single human baseline.

Known issues & criticisms

  • The public static data can leak into training.
  • Some synthetic tasks test retrieval or procedural simulation more than document understanding.
  • Very long examples make model truncation policies a major confounder.

Contamination notes

Risk is high for published splits. Synthetic task families can be regenerated for lower leakage, but leaderboard-style claims on the released files should assume contamination is possible.

How to run it

git clone https://github.com/OpenBMB/InfiniteBench.git
cd InfiniteBench
bash scripts/download_dataset.sh
pip install -r requirements.txt

Then run a task from src:

python eval_gpt4.py --task longbook_qa_eng

History & lineage

InfiniteBench was submitted in February 2024 and published at ACL 2024. It followed earlier 5k-15k long-context suites such as LongBench by making 100k+ token contexts the central stress condition.

Further reading