benchwiki
long-context

Needle-in-a-Haystack

Saturated

Synthetic retrieval stress test that hides facts at different depths and context lengths, then asks the model to recover them.

last updated 2026-08-12

Performance over time

No performance time series — this benchmark has no cleanly-sourced, metric-consistent aggregate scores to plot. See the prose below for why, and the facts sidebar for what it measures.

Overview

Needle-in-a-Haystack is the canonical long-context retrieval smoke test. It inserts a small target fact, UUID, or chain of facts into a long distractor context and asks the model to retrieve it. Results are normally plotted over two axes: total context length and the depth where the needle was inserted.

Its strength is simplicity. Its weakness is also simplicity: retrieving one unique fact from synthetic noise is not the same as understanding a long document.

Task details

The modern niah CLI supports sweeps over context lengths, depth percentages, seeds, providers, and task types. Built-in tasks include a single fact, multiple facts, UUID retrieval, and UUID-chain hops.

Example

A run may insert a UUID-associated fact halfway through a 32k-token context and ask for the value associated with that UUID. If the response repeats the exact expected value, the cell passes.

Scoring methodology

Basic NIAH uses exact-match scoring for the inserted fact. Multi-needle and UUID-chain variants can use fractional scores, such as recovered hops out of a chain. Report the grid, not just the mean, because positional failures are the point of the test.

What it doesn’t measure

NIAH does not measure summarization, multi-document reasoning, discourse understanding, or factuality. A model can pass vanilla NIAH and still fail long-context reasoning tasks that require combining evidence.

Known issues & criticisms

  • Vanilla NIAH is too easy for many modern long-context models.
  • The inserted needle is often distributionally obvious.
  • A single average hides where in the context the model failed.

Contamination notes

Risk is low when needles and seeds are generated fresh for each run. Risk rises if a published grid, fixed haystack, or fixed inserted sentence is reused.

How to run it

pip install needlehaystack
niah demo --fake

For a real provider-backed run:

niah run configs/runs/single_needle.example.yaml

History & lineage

Greg Kamradt popularized NIAH in late 2023 while pressure-testing GPT-4-128K and Claude 2.1. Later benchmarks such as RULER and OpenAI-MRCR generalize the idea with multiple needles, harder distractors, aggregation, and multi-hop variants.

Further reading