benchwiki
reasoning

BIG-Bench Hard

Saturated
2022-10-17 → today

23 BIG-Bench tasks where models trailed humans — the benchmark that proved chain-of-thought prompting works, then fell to the reasoning-model era.

last updated 2026-07-25

Performance over time

vendor-reported independent
60708090100202220232024human baseline 67.7%

Baseline: Average human-rater performance across the 23 tasks in the original paper; the best (max) human rater averaged ~94.4%. BBH tasks were selected precisely as the BIG-Bench tasks where prior models fell below the 67.7% average-rater line. (Canonical Table 1 values from the paper.)

Overview

BIG-Bench Hard is the benchmark that made chain-of-thought prompting canonical. In October 2022, Suzgun et al. took the parent BIG-Bench suite and pulled out the 23 tasks (6,511 examples) on which prior language models had failed to beat the average human rater — the genuinely hard residue of a sprawling suite. Then they asked a simple question that reshaped prompting practice: what happens if you let the model reason step by step before answering?

The result was dramatic. With 3-shot chain-of-thought (CoT) prompting, models leapt: Codex (code-davinci-002) surpassed the average human rater on 17 of the 23 tasks, and PaLM 540B on 10 of 23 — while the same models with direct, answer-only prompting did not. CoT wasn’t a marginal trick; on these tasks it was the difference between sub-human and super-human. BBH became the standard evidence that eliciting reasoning unlocks latent capability, and “3-shot CoT on BBH” became a fixture of model reports for the next two years.

BBH is now saturated. The reasoning-model era pushed frontier systems past ~90% on the 23-task average and to near-perfect on many individual tasks, collapsing the headroom that made it useful. In 2025 Google DeepMind released BBEH (BIG-Bench Extra Hard), replacing each BBH task with a harder counterpart on which even the best general-purpose models score below 10% — an explicit acknowledgment that BBH had stopped discriminating. The benchmark that proved reasoning prompting works was, fittingly, retired by models that reason.

Task details

The 23 tasks are multi-step reasoning problems drawn from BIG-Bench: arithmetic and logical word problems, geometric and temporal reasoning, tracking shuffled objects, disambiguation, sarcasm and language understanding, and formal deduction. Each has a discrete, checkable answer (multiple-choice or short exact-match), which is what makes the suite cleanly gradable despite the reasoning involved.

  • Input: three worked few-shot examples (with or without reasoning traces, depending on the mode), then the target question.
  • Output: for CoT mode, a reasoning trace followed by a final answer, from which the answer is parsed; for direct mode, the answer alone.
  • Aggregate: the unweighted mean accuracy across the 23 tasks.

Example

From the navigate task (spatial/temporal reasoning about whether a sequence of moves returns to the start):

Question: If you follow these instructions, do you return to the starting point? “Take 8 steps. Turn around. Take 8 steps.”

Direct answer (often wrong without reasoning): “No.”

Chain-of-thought: “Start at 0. Take 8 steps → at +8. Turn around (now facing back). Take 8 steps → back at 0. I end where I started.” → “Yes.”

This is the BBH pattern in miniature: the answer is trivial if the model works through the intermediate state, and error-prone if it guesses in one shot. Aggregated over 23 such tasks, that gap is exactly what CoT prompting closed.

Scoring methodology

Accuracy by exact match on the parsed final answer, averaged unweighted across the 23 tasks — a small task and a large task count equally. The reporting subtlety that matters most here is the direct-vs-CoT dual convention:

  • Direct (answer-only): the model outputs the answer with no reasoning. This is the harder setting and the one that stays low.
  • Chain-of-thought (3-shot): the model reasons before answering. This is the headline setting and the one that saturated.

A “BBH score” is meaningless without saying which — the two differ by tens of points for the same model, and BBH’s entire contribution was documenting that gap. Modern reasoning models blur the line further by producing CoT-style traces even when prompted “directly,” which is part of why the benchmark no longer discriminates. Always report shots and whether CoT was used.

What it doesn’t measure

BBH measures multi-step reasoning on problems with short, checkable answers — not open-ended reasoning, not long-horizon planning, not tool use, and not the quality of the reasoning trace itself (only whether the final answer is right, so a correct answer from flawed reasoning still scores). Because its tasks are inherited from BIG-Bench and public since 2022, a high score also increasingly reflects exposure rather than reasoning (see Contamination). It is English-only and, now that it is saturated, says almost nothing about frontier models — for those, its successor BBEH or a current reasoning benchmark is the right tool.

Known issues & criticisms

  • Saturation collapses the signal. With frontier models above ~90% on the 23-task average and near-perfect on many tasks, differences at the top are within noise; the aggregate no longer ranks strong models.
  • Contamination. BBH inherits BIG-Bench’s public tasks and its ignored canary, and its own popularity made it a prime candidate for inclusion in training and instruction-tuning data. Post-2022 scores should be read as an upper bound on true reasoning ability.
  • Direct-vs-CoT non-comparability. Scores depend heavily on whether CoT was used and how the final answer is parsed from a reasoning trace; reasoning-tuned models that always “think” make the direct/CoT distinction ill-defined, so cross-model comparisons need care.
  • Unweighted task averaging. The macro average lets a small, easy task move the aggregate as much as a large, hard one — so the headline number can be gamed by the task mix.
  • Answer-parsing fragility. Extracting the final answer from free-form CoT output is heuristic; parsing failures can understate a model that reasoned correctly but formatted its answer unexpectedly.

Contamination notes

Risk: high. BBH is a subset of BIG-Bench, so it inherits both the public availability of those tasks and the canary-string guard that trainers largely ignored — see the parent page for the full canary story. What makes BBH’s exposure worse than the parent’s is its own success: because “3-shot CoT on BBH” appeared in so many papers and repos, the tasks and even worked solutions circulate widely, and instruction-tuning mixes have picked them up. There is no refreshed or held-out BBH. This unfixable contamination, together with saturation, is exactly why Google DeepMind built BBEH from fresh tasks rather than patching BBH.

How to run it

lm-evaluation-harness (CoT few-shot is the standard BBH configuration):

pip install lm-eval
lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-3.1-8B \
  --tasks bbh_cot_fewshot \
  --batch_size auto
# answer-only (direct) BBH is exposed as bbh_fewshot; per-task variants also exist

Official BIG-Bench Hard repository (canonical tasks + the paper’s CoT prompts):

git clone https://github.com/suzgunmirac/BIG-Bench-Hard
# bbh/ holds the 23 task files; cot-prompts/ holds the 3-shot CoT exemplars

Always report shots, whether CoT was used, and the answer-extraction method — a bare “BBH 88” is not a reproducible measurement.

History & lineage

  • 2022-06 — Parent suite BIG-Bench released; too broad and uneven to serve as a sharp evaluation.
  • 2022-10 — BBH distills the 23 tasks models were worst at. With 3-shot CoT, Codex beats the average human rater on 17/23 tasks and PaLM on 10/23, while direct prompting does not — establishing chain-of-thought as a canonical technique.
  • 2023–2024 — “3-shot CoT on BBH” becomes a standard line in model reports; instruction-tuned and then reasoning-tuned models climb toward and past 90% on the aggregate.
  • 2024 — BBH is treated as saturated: near-ceiling frontier scores and pervasive contamination end its usefulness at the top.
  • 2025 — Google DeepMind releases BBEH (BIG-Bench Extra Hard) (arXiv:2502.19187), rebuilding each of the 23 tasks at much higher difficulty (best general-purpose models ≈ 9.8%) as the designated successor. Its page isn’t written here yet — hence this page’s successor slug does not yet resolve to a link.

If you’re choosing a reasoning benchmark today: use BBEH or a current frontier reasoning eval; keep BBH only as a historical reference and a cheap regression check for smaller models.

Further reading