benchwiki
maths

GSM-Hard

Deprecated

GSM8K test problems with unusually large numerical values, designed to separate reasoning from easy arithmetic.

last updated 2026-07-29

Performance over time

No performance time series — this benchmark has no cleanly-sourced, metric-consistent aggregate scores to plot. See the prose below for why, and the facts sidebar for what it measures.

Overview

GSM-Hard is a numerical-robustness variant of GSM8K released by the authors of PAL (Program-Aided Language Models). It keeps the language and operation structure of GSM8K’s 1,319 test questions but replaces ordinary numbers with larger, less common values. The intended diagnostic is narrow: can a system preserve the correct reasoning program when mental arithmetic and memorized answer patterns become unreliable?

That diagnostic is useful, especially when comparing chain-of-thought with Python-backed program execution. It is not a clean harder curriculum. Automatic substitution can make quantities absurd, produce negative counts or prices, and leave stale numbers inside prose or code comments. This page therefore marks GSM-Hard deprecated as a standalone benchmark while retaining it as a reproducible stress test.

Task details

Each record contains an input question, a Python code solution, and a numeric target. The released Hugging Face dataset has one split named train, but its 1,319 rows correspond to transformed GSM8K test items and are normally used for evaluation. A model returns a final number directly or generates a program whose execution returns that number.

Example

The official dataset viewer includes a robe problem in which the original value of two blue-fiber bolts is replaced by 2,287,720. The relation “half that much white fiber” is unchanged, so the expected total becomes 3,431,580. This illustrates the intended arithmetic scaling. Other public rows show the failure mode: a replacement can exceed a stated daily total and make the computed remainder negative.

Scoring methodology

Accuracy is the fraction of extracted final numeric answers equal to the public target. PAL instead asks the model to generate Python and executes solution(); this reduces arithmetic errors but means a PAL score measures a model-plus-runtime system. Direct chain-of-thought, calculator, and Python-backed results are not interchangeable.

What it doesn’t measure

GSM-Hard does not add new mathematical concepts, deeper dependency graphs, proofs, multilingual reasoning, or realistic large-number applications. Because it preserves GSM8K templates, success can reflect template familiarity rather than generalization to genuinely new problems.

Known issues & criticisms

The principal issue is visible in the official data itself: mechanical number replacement can violate real-world constraints and yield nonsensical negative quantities. The dataset card documents only limited construction detail, and no transformed-set human validation rate is reported. The set also has no conventional train/dev/test organization despite the split label train.

Contamination notes

Risk is high. GSM8K is a widely circulated static benchmark, and GSM-Hard’s complete questions, targets, and executable solutions are public. Replacing numerals changes exact strings but not underlying templates. It is therefore unsuitable as evidence against memorization or training-data overlap.

How to run it

Load the official release directly:

from datasets import load_dataset
data = load_dataset("reasoning-machines/gsm-hard", split="train")

For PAL-style evaluation, clone the reference repository, generate a solution() function, execute it in a sandbox, and compare its return value with target. Record whether code execution was enabled and pin the dataset revision; a public typo was corrected in 2024.

History & lineage

GSM-Hard descends directly from GSM8K and was announced in the PAL repository in January 2023. It should be understood alongside newer robustness approaches that generate semantically checked or parameterized variants, rather than as a successor that replaces GSM8K.

Further reading