BIG-Bench

Overview

BIG-Bench — the Beyond the Imitation Game Benchmark — was the field’s great collaborative attempt to ask a language model everything at once. Released in June 2022 and assembled by roughly 450 authors across 132 institutions, it is not a single test but a sprawling suite of 204 tasks (plus programmatic ones) covering linguistics, mathematics, commonsense, world knowledge, code, social bias, and dozens of non-English languages. Anyone could contribute a task, and they did — the result is equal parts serious evaluation and community time-capsule, ranging from rigorous reasoning probes to jokes, puzzles, and deliberately strange challenges.

Its ambition was to measure capabilities that individual benchmarks missed, and to extrapolate — to see where scaling curves were heading. For a moment in 2022 it was the broadest snapshot of what large models could and couldn’t do. But breadth came at the cost of coherence: tasks used wildly different formats and metrics, quality varied from excellent to broken, and the single aggregate score papered over all of it.

BIG-Bench is now deprecated — and deliberately labelled that way rather than “saturated.” Models did not uniformly max it out; rather, it stopped being used as an evaluation. Its famous “canary” contamination guard failed in practice (see Contamination), its uneven task quality made the aggregate hard to trust, and the community distilled the parts that mattered into sharper successors. The most important of these, BIG-Bench Hard, took the 23 tasks models were worst at and became far more influential than the parent suite ever was. BIG-Bench’s legacy is less a scoreboard than a seedbed.

Task details

Each task ships as a self-contained unit — a task.json (or a programmatic Python task) with its inputs, targets, and a declared preferred metric. Formats are heterogeneous by design:

• Multiple-choice tasks (the plurality) scored by which option the model ranks highest.
• Exact-match / generation tasks scored against reference targets, sometimes with BLEU/ROUGE-style similarity.
• Custom scorers for tasks that need bespoke logic.

To produce a single number, per-task scores are normalized to a common scale (roughly 0 = poor, 100 = an expert-rubric ceiling) and averaged. Two curated subsets matter in practice: BIG-bench Lite (24 tasks chosen to be cheap and representative, the usual entry point) and BIG-Bench Hard (the 23 hardest-for-models tasks, which became a benchmark in its own right).

Example

BIG-Bench’s range is the point, so two tasks from opposite ends illustrate it better than one:

formal_fallacies (deductive logic): given a short argument, decide whether the conclusion follows deductively from the premises. A reasoning probe with clean right/wrong answers.

emoji_movie (commonsense/culture): map a string of emoji to the movie it represents (🦁👑 → The Lion King). A playful task contributed for fun that nonetheless tests grounded cultural association.

The same suite grades a model on formal logic and on emoji charades, then averages them. That juxtaposition is exactly why the aggregate is treated with suspicion and why the community pulled the serious reasoning tasks out into BBH.

Scoring methodology

There is no single scoring rule — which is why this page’s metric.scoring is composite. Each task’s preferred metric produces a raw score; those are normalized per task and averaged into the suite aggregate. The subtleties that make BIG-Bench numbers hard to compare:

• Task weighting. The unweighted average gives a 100-example joke task the same influence as a 3,000-example reasoning task. Different papers include different task subsets, so “the BIG-Bench score” is rarely the same measurement twice.
• Normalization choices. How each task’s raw metric maps onto the 0–100 scale is a modeling decision that affects the aggregate.
• Subset ambiguity. “BIG-Bench” might mean the full suite, BIG-bench Lite, or a hand-picked slice. Always state which.

What it doesn’t measure

Paradoxically for a suite designed to measure everything, BIG-Bench measures no one thing well. A single aggregate cannot tell you whether a model is good at reasoning, knowledge, or code — only at the particular, uneven basket of 204 tasks someone happened to contribute. It is a breadth instrument, not a precision one: excellent for a rough capability snapshot circa 2022, poor for the fine, trustworthy comparison this site cares about. For that, use a focused benchmark (or BIG-Bench Hard’s reasoning slice) rather than the parent aggregate.

Known issues & criticisms

• The canary that everyone ignored. BIG-Bench’s headline contamination defense — a GUID string embedded in every task file, which trainers were asked to detect and exclude — depended entirely on voluntary compliance. It was widely ignored, and BIG-bench tasks are now pervasive in web-scale training corpora. It remains the field’s clearest cautionary tale that opt-in decontamination does not work.
• Wildly uneven task quality. Because anyone could contribute, tasks range from rigorous to broken — mislabeled targets, ambiguous prompts, and joke-adjacent content sit alongside serious evaluations. The suite has no uniform quality bar.
• Aggregate sensitivity to weighting. The suite-level number swings with task selection and weighting (see Scoring methodology), so cross-paper comparisons of “BIG-Bench score” are unreliable.
• Disputed and non-serious content. Several tasks are puzzles, in-jokes, or of contested validity, diluting the aggregate as a capability signal.

Contamination notes

Risk: high, and instructive. BIG-Bench is the origin of the “canary string” idea — a fixed GUID placed in every task file so that anyone training a model can grep their corpus for it and exclude the benchmark. The mechanism is elegant and completely dependent on trainers actually honoring it. They largely did not: the tasks are MIT/Apache-licensed, public since 2022, and mirrored across the web, and the canary did nothing to stop them from being scraped. The lesson propagated forward — later benchmarks lean on private test sets or refresh cycles precisely because BIG-Bench proved that a “please don’t train on this” marker is not a control. Treat any BIG-Bench score from a modern model as memorization-inclusive.

How to run it

lm-evaluation-harness (BIG-bench tasks are integrated):

pip install lm-eval
lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-3.1-8B \
  --tasks bigbench \
  --batch_size auto
# individual tasks are exposed as bigbench_<task_name>; BIG-bench Lite as a group

Official BIG-bench repository (canonical task definitions, incl. programmatic tasks):

git clone https://github.com/google/BIG-bench
# do NOT strip the canary GUID from task files

Report which subset you ran (full / Lite / a named slice) and the weighting — a bare “BIG-Bench score” is uninterpretable without both.

History & lineage

• 2022-06 — BIG-Bench released (arXiv v1, later TMLR 2023): 204 tasks, ~450 authors, 132 institutions. The most ambitious single snapshot of LM capabilities to date, and the debut of the canary-string contamination guard.
• 2022-10 — BIG-Bench Hard distilled the 23 tasks where models trailed the average human rater. It promptly eclipsed its parent by becoming the benchmark on which chain-of-thought prompting was shown to work.
• 2023–2024 — Canary leakage, uneven quality, and the rise of focused successors erode BIG-Bench’s use as a live evaluation; the aggregate falls out of launch posts.
• 2025 — Google DeepMind’s BBEH (BIG-Bench Extra Hard) rebuilds BBH’s tasks at much higher difficulty, continuing the lineage past its saturated child.

The through-line: a community mega-suite → a hard distillate (BBH) that defined the CoT era → a fresh, harder successor (BBEH) once the distillate saturated.

Further reading

• Srivastava et al., 2022 — original paper: https://arxiv.org/abs/2206.04615
• google/BIG-bench — the task repository and canary documentation: https://github.com/google/BIG-bench
• Suzgun et al., 2022 — the BBH distillate: https://arxiv.org/abs/2210.09261