GSM8K

Overview

GSM8K is the benchmark that taught the field how good grade-school arithmetic could be at exposing what language models can and can’t do. Its ~8,500 problems are the kind a bright middle-schooler handles — “Natalia sold clips to 48 friends in April, then half as many in May; how many did she sell altogether?” — each solvable in 2–8 steps of basic arithmetic (+ − × ÷), written and solved in natural language by human authors.

A historical note that reframes the benchmark: the paper’s actual contribution was training verifiers — a separate model that scores candidate solutions, letting you sample many and keep the best. GSM8K was the dataset built to study that method, and the paper’s headline was that a 6B model plus a verifier could rival a fine-tuned 175B one (“a boost approximately equivalent to a 30× model size increase”). The verifier idea outlived its original framing: it is a direct ancestor of the process-reward models and RLHF-for-reasoning work that powers today’s reasoning models. But the dataset outlived the method’s framing even more thoroughly — GSM8K became the default grade-school-math yardstick, cited far more often than the verifier technique it was built to evaluate.

Its second claim to fame is chain-of-thought. On GSM8K, letting a model reason step-by-step before answering is not a marginal trick — it is the difference between near-random and competent. PaLM 540B went from 17.9% with standard prompting to 56.9% with chain-of-thought, and self-consistency (sampling many chains, taking the majority vote) pushed it to 74.4%. GSM8K is, alongside BIG-Bench Hard, one of the two landmark demonstrations that eliciting reasoning unlocks latent capability — the CoT story this wiki tells twice, on two benchmarks, because it reshaped how everyone prompts.

GSM8K is now saturated — and it is the wiki’s single clearest illustration of why saturated does not mean solved. Frontier models score 97–99%, but Apple’s GSM-Symbolic work showed that changing only the names and numbers, and especially inserting one irrelevant-but-plausible clause, collapses accuracy across models (see Known issues). The number is at the ceiling; the underlying reasoning is not robust. Hold both facts at once — that tension is the most important thing on this page.

Task details

GSM8K ships as 7,473 training + 1,319 test problems (the paper rounds these to “7.5K / 1K”). Each problem is a short natural-language word problem with a human-written, step-by-step solution ending in a delimiter:

Natalia sold 48 clips in April, and half as many in May.
How many clips did she sell altogether?
She sold 48 / 2 = 24 clips in May.
Altogether she sold 48 + 24 = 72 clips.
#### 72

Input: the problem text (in the standard setup, preceded by a few CoT exemplars).
Output: a reasoning trace and a final answer; only the number after #### is graded.
Aggregate: exact-match accuracy over the 1,319 test problems.

The HuggingFace dataset (openai/gsm8k) has two configs: main (problem + solution) and socratic, which interleaves guiding sub-questions (“How many clips did she sell in May?”) before each step — a structure aimed at studying process, not just the final answer.

dataset.size is 1,319 (the test set — the reported configuration) per the size-matches-primary convention (see GPQA); the 7,473 training problems exist to fine-tune on, not to score.

Example

The Natalia problem above is the canonical GSM8K item (it is literally the first test example). Its shape is the whole benchmark in miniature: no advanced math, just multi-step bookkeeping — parse the quantities, apply two operations in the right order, and don’t drop a term. The failure modes GSM8K catches in weaker models are arithmetic slips and losing track of intermediate state across steps, which is exactly why chain-of-thought (writing the intermediate 48 / 2 = 24 down) helps so much: it externalizes the state the model would otherwise have to hold implicitly.

Scoring methodology

Scoring is exact-match on the final number — nothing else. This has a crucial consequence: GSM8K measures answer-correctness, not process-correctness. A model can reach 72 through valid reasoning, through flawed reasoning that happens to cancel out, or through memorization; all score identically. Conversely, a correct method with one arithmetic slip scores zero. So a GSM8K number tells you how often the final answer is right, not how reliable the reasoning is — precisely the gap the socratic config and the later process-reward line of work exist to close.

Three reporting traps to state explicitly:

Direct vs chain-of-thought. CoT is worth tens of points here (PaLM 540B: 17.9 → 56.9). A “GSM8K” number is meaningless without knowing whether the model reasoned; the two settings are not comparable.
maj@k voting. Some headline numbers use self-consistency / majority vote over k samples (PaLM 540B: 56.9 → 74.4 at CoT → CoT+SC). maj@k is not maj@1; report k.
Shots. Originally reported at 8-shot CoT; modern instruct-model reporting is often 0-shot or 5-shot CoT. Different shot counts give different numbers. This page’s headline axis is CoT accuracy; state shots and k with any figure.

What it doesn’t measure

GSM8K measures elementary multi-step arithmetic reasoning in English — and nothing harder. It says nothing about competition or research mathematics: no algebra-proper, geometry proofs, number theory, or olympiad problems, which is the province of MATH, AIME, and FrontierMath. It does not measure the robustness of reasoning (GSM-Symbolic shows the headline number hides brittleness), nor process-correctness (only the final number is graded), nor tool-assisted math (it is closed-book by convention). And now that it is saturated, it says almost nothing about frontier models except as a cheap regression check for small ones. A high GSM8K score means “usually gets grade-school word problems right,” not “can do math.”

Known issues & criticisms

GSM-Symbolic: the score is saturated, the capability is not. Mirzadeh et al. (Apple, 2024, arXiv:2410.05229, ICLR 2025) regenerated GSM8K problems from templates, changing only names and numbers, and saw notable accuracy drops — evidence that models partly pattern-match the specific public problems. Their GSM-NoOp variant added a single irrelevant-but-plausible clause (e.g. mentioning some fruit are “a bit smaller than average”) and accuracy fell by up to ~65% across models, because they folded the irrelevant number into the arithmetic. This is the wiki’s best concrete proof that a saturated score can coexist with fragile reasoning — read it directly against this page’s status_evidence.
Label/solution errors. GSM8K-Platinum (Vendrow, Vendrow, Beery, Madry / MIT, 2025; madrylab/gsm8k-platinum) manually revised the full test set, fixing mislabeled answers and removing ambiguous problems, and found that some apparent “model failures” were actually benchmark errors — the same forensic lesson as MMLU-Redux. On the cleaned set, frontier reasoning models make only a handful of errors, sharpening comparisons the noisy original blurred.
Contamination via worked solutions. The problems and their full solutions have been public since 2021 (see Contamination), so high scores are memorization-inclusive.
Exact-match masks reasoning errors. Grading only the final number credits right-answer/wrong-reasoning and penalizes right-reasoning/wrong-arithmetic (see Scoring methodology).
Grade-school scope. GSM8K says nothing about harder math; a strong score is easily over-read as “good at mathematics.”

Contamination notes

Risk: high. GSM8K is public, MIT-licensed, and everywhere — including on sites that reproduce its natural-language solutions in full, which makes it unusually easy for a model to have seen not just the questions but the worked answers. It is fully static with no refresh cycle. This is the same structural situation that saturated MMLU, and it is the direct motivation for the variant ecosystem: GSM-Symbolic perturbs the surface form to detect memorization, GSM8K-Platinum cleans the labels, and GSM-Plus / GSM-Hard stress the numbers and adversarial structure. If you need a grade-school-math signal you trust on a modern model, use one of those perturbed/cleaned variants rather than the original, and treat any near-ceiling original-GSM8K score as an upper bound on genuine reasoning.

How to run it

lm-evaluation-harness (community standard; CoT is the meaningful setting):

pip install lm-eval
lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-3.1-8B \
  --tasks gsm8k \
  --batch_size auto
# gsm8k_cot exposes explicit chain-of-thought prompting; report shots and any maj@k

Official OpenAI repo / dataset:

from datasets import load_dataset
ds = load_dataset("openai/gsm8k", "main")     # or "socratic"
print(ds["test"][0])   # the Natalia problem; answer is the text after "####"

Report shots, whether CoT was used, and any maj@k — and, given contamination, consider running GSM8K-Platinum or GSM-Symbolic alongside to check the number is real.

History & lineage

2021-10 — GSM8K released (Cobbe et al., arXiv:2110.14168) to study verifiers; the dataset becomes the default grade-school-math benchmark. Fine-tuned GPT-3 175B sits around ~35%.
2022 — Chain-of-thought (Wei et al., arXiv:2201.11903) takes PaLM 540B from 17.9% to 56.9% on GSM8K; self-consistency (Wang et al., arXiv:2203.11171) reaches 74.4%. GSM8K becomes the canonical CoT demonstration, alongside BIG-Bench Hard.
2023 — GPT-4 reports 92.0% (5-shot CoT); GSM8K is nearly solved as a frontier benchmark.
2024 — Frontier and reasoning-tuned models exceed 97%; GSM8K is treated as saturated. GSM-Symbolic (Apple) shows the saturation is partly illusory — trivial perturbations and irrelevant clauses collapse accuracy, evidence of pattern-matching over robust reasoning.
2025 — GSM8K-Platinum cleans the test set, revealing that some residual “failures” were label errors and sharpening frontier comparisons.

GSM8K has no single successor (its successor is null): MATH, AIME, and FrontierMath are harder math benchmarks from different lineages and teams, not designated replacements, and the GSM-* variants are diagnostic tools, not heirs. For genuine headroom today, climb the difficulty ladder MATH → AIME → FrontierMath; keep GSM8K as a cheap regression check and a teaching example.

Performance over time