Overview
RULER asks a direct question: what is the real usable context size of a long-context model? It starts from Needle-in-a-Haystack but expands the test into configurable synthetic tasks with multiple needles, varied needle types, multi-hop tracing, and aggregation. The paper reports that models with nominal 32k+ context windows can still drop sharply as length and task complexity increase.
Task details
RULER generates synthetic contexts at chosen lengths and evaluates a set of representative tasks. The original paper evaluates 17 long-context models on 13 tasks. The key dimensions are sequence length and task complexity, not a fixed natural-language dataset.
Example
A simple RULER item may hide several facts in a long context and ask for one of them. A harder item may require following a chain or aggregating information scattered through the prompt.
Scoring methodology
RULER mostly uses exact answer accuracy, averaged over generated tasks and lengths. The most important report is a curve: accuracy as context length grows. A model that is perfect at 8k but collapses at 64k has not demonstrated a 64k effective context length.
Current README stats
The visual graph on this page shows only the Avg. statistic from the official NVIDIA RULER README snapshot checked on 2026-08-12. That Avg. is the mean accuracy across the 4K, 8K, 16K, 32K, 64K, and 128K result columns for the selected high-scoring rows below. The graph does not show the claimed context length, effective context length, per-length cells, wAvg. (inc), or wAvg. (dec).
| Model | Claimed length | Effective length | 4K | 8K | 16K | 32K | 64K | 128K | Avg. shown in graph |
|---|---|---|---|---|---|---|---|---|---|
| Jamba-1.5-large | 256K | >128K | 96.7 | 96.6 | 96.4 | 96.0 | 95.4 | 95.1 | 96.0 |
| Gemini-1.5-pro | 1M | >128K | 96.7 | 95.8 | 96.0 | 95.9 | 95.9 | 94.4 | 95.8 |
| Qwen2.5-14B-Instruct-1M | 1M | >128K | 97.5 | 97.1 | 94.6 | 94.9 | 94.9 | 92.2 | 95.7 |
| Qwen3-235B-A22B | 128K | >128K | 97.7 | 97.2 | 96.4 | 95.1 | 93.3 | 90.6 | 95.0 |
| Qwen3-14B | 128K | >128K | 98.0 | 97.8 | 96.4 | 96.1 | 94.0 | 85.1 | 94.6 |
| Jamba-1.5-mini | 256K | >128K | 95.6 | 95.6 | 94.8 | 94.6 | 92.8 | 90.0 | 93.9 |
| Qwen3-32B | 128K | >128K | 98.4 | 96.0 | 96.2 | 94.4 | 91.8 | 85.6 | 93.7 |
| EXAONE-4.0-32B | 128K | >128K | 96.3 | 94.9 | 93.9 | 93.6 | 91.7 | 88.2 | 93.1 |
| Qwen2.5-7B-Instruct-1M | 1M | >128K | 96.8 | 95.3 | 93.0 | 91.1 | 90.4 | 84.4 | 91.8 |
| GPT-4-1106-preview | 128K | 64K | 96.6 | 96.3 | 95.2 | 93.2 | 87.0 | 81.2 | 91.6 |
Source: NVIDIA/RULER README.
What it doesn’t measure
RULER does not measure natural-document quality, summarization, legal reasoning, or citation faithfulness. It is a synthetic diagnostic for retrieval, tracing, and aggregation robustness.
Known issues & criticisms
- Synthetic tasks can be narrower than real document understanding.
- Generator settings and seeds affect difficulty.
- A single aggregate score hides the effective-context curve.
Contamination notes
Risk is low when new configurations and seeds are used. Because the generator and default configs are public, published static configurations should not be treated as private tests.
How to run it
git clone https://github.com/NVIDIA/RULER.git
cd RULERFollow the official repository instructions for model configuration, sequence length, and task selection. Report the generated task set and length grid with any score.
History & lineage
RULER was submitted in April 2024 and accepted at COLM 2024. It explicitly positions vanilla Needle-in-a-Haystack as too superficial and extends it into a broader synthetic long-context evaluation.
Further reading
- RULER paper: https://arxiv.org/abs/2404.06654
- Official RULER repository: https://github.com/NVIDIA/RULER
- Needle-in-a-Haystack repository: https://github.com/gkamradt/needle-in-a-haystack