Overview
OpenAI-MRCR is a long-context multiple-needle benchmark. MRCR stands for multi-round co-reference resolution. The model reads a long synthetic conversation containing repeated similar user requests and assistant responses, then must return a specific instance, such as the second poem about a topic. The repeated asks are the needles; the surrounding synthetic conversation is the haystack.
The task is harder than a single unique needle because the target is not just “find the only fact.” The model has to distinguish same-distribution distractors, count occurrences, and return the requested instance.
Task details
The dataset card defines 2-, 4-, and 8-needle variants. The conversation is synthetically generated, with 438 distinct entities and 10 writing formats. There are 100 samples per token bin, and the bins range from 4k-8k tokens up to 524k-1,048k tokens.
Example
A prompt may include several requests for poems, blog posts, or social media posts about different entities. At the end, the model is asked to prepend a random alphanumeric hash to the second instance of a particular requested writing. If it returns the wrong occurrence, the answer should score poorly.
Scoring methodology
The metric is a difflib.SequenceMatcher ratio. The requested hash must appear at the beginning of the answer; if it is missing, the match ratio is set to zero. If the hash is correct, the sampled answer is stripped and compared to the stripped ground-truth answer.
MRCR v2 leaderboard stats
This page is now using MRCR v2 (8-needle) as the plotted benchmark version. LLM Stats describes MRCR v2 (8-needle) as the 8-needle Multi-Round Coreference Resolution variant, scored by mean_similarity on a raw 0-1 scale where higher is better. The LLM Stats snapshot used here was last updated on 2026-07-22, covers 21 models, and is marked as 21 self-reported / 0 verified results.
The visual graph shows only the LLM Stats MRCR v2 (8-needle) score, converted to percent for the y-axis. It does not show per-context-bin scores, 2-needle or 4-needle variants, context window, cost, model size, license, or verification status.
| Rank | Model | Vendor | Raw LLM Stats score | Score shown in graph | Context | Notes |
|---|---|---|---|---|---|---|
| 1 | GPT-5.6 Sol | OpenAI | 0.915 | 91.5% | 1.1M | LLM Stats leader |
| 2 | GPT-5.6 Terra | OpenAI | 0.896 | 89.6% | 1.1M | Cheapest within 10% of leader per LLM Stats FAQ |
| 3 | Claude Opus 4.6 | Anthropic | 0.760 | 76.0% | 1.0M | Proprietary |
| 4 | GPT-5.5 | OpenAI | 0.740 | 74.0% | 1.1M | Proprietary |
| 5 | Gemma 4 31B | 0.664 | 66.4% | 262K | Top open model on LLM Stats | |
| 6 | Gemini 3.1 Flash-Lite | 0.601 | 60.1% | 1.0M | Proprietary | |
| 7 | Gemini 3.6 Flash | 0.540 | 54.0% | 1.0M | Proprietary | |
| 8 | Gemma 4 26B-A4B | 0.441 | 44.1% | 262K | Open model | |
| 9 | Gemma 4 12B | 0.434 | 43.4% | - | Open model | |
| 10 | GPT-5.6 Luna | OpenAI | 0.413 | 41.3% | 1.1M | Proprietary |
| 11 | GPT-5.4 mini | OpenAI | 0.336 | 33.6% | 400K | Proprietary |
| 12 | GPT-5.4 nano | OpenAI | 0.331 | 33.1% | 400K | Proprietary |
| 13 | Gemini 3.5 Flash | 0.266 | 26.6% | 1.0M | Proprietary | |
| 14 | Gemini 3.1 Pro | 0.263 | 26.3% | 1.0M | Proprietary | |
| 14 | Gemini 3 Pro | 0.263 | 26.3% | - | Proprietary | |
| 16 | Gemma 4 E4B | 0.254 | 25.4% | - | Open model | |
| 17 | Gemini 3 Flash | 0.221 | 22.1% | 1.0M | Proprietary | |
| 18 | Gemini 3.5 Flash-Lite | 0.213 | 21.3% | 1.0M | Proprietary | |
| 19 | Gemma 4 E2B | 0.191 | 19.1% | - | Open model | |
| 20 | Gemini 2.5 Pro Preview 06-05 | 0.164 | 16.4% | - | Proprietary | |
| 21 | Gemma 3 27B | 0.135 | 13.5% | - | Open model |
Source: LLM Stats MRCR v2 (8-needle) leaderboard. Background: Michelangelo / LSQ paper and OpenAI MRCR dataset card.
What it doesn’t measure
OpenAI-MRCR does not measure natural document reasoning, summarization, or factuality. It isolates order-sensitive retrieval under long synthetic conversational context.
Known issues & criticisms
- The data is public and synthetic.
- Sequence similarity can partially credit text that is not semantically correct.
- Results depend heavily on context-length bin and needle count.
Contamination notes
Risk is high for the public Hugging Face split. The task family is still useful when regenerated privately, but the released data should not be treated as unseen for post-release models.
How to run it
from datasets import load_dataset
dataset = load_dataset("openai/mrcr")Report the needle variant, token bin, prompt template, hash rule, and SequenceMatcher implementation with any result.
History & lineage
OpenAI-MRCR is inspired by earlier multiple-needle long-context evaluations, including MRCR-style tests introduced in Gemini reporting. It extends the Needle-in-a-Haystack idea by requiring co-reference and occurrence selection among repeated similar requests.
Further reading
- OpenAI MRCR dataset card: https://huggingface.co/datasets/openai/mrcr
- HELM Long Context leaderboard using OpenAI MRCR: https://crfm.stanford.edu/helm/long-context/latest/
- RULER paper for related synthetic long-context stress testing: https://arxiv.org/abs/2404.06654