benchwiki
maths

MGSM

Nearing saturation
2022-10-06 → today

A human-translated 250-problem GSM8K subset for comparing grade-school mathematical reasoning across ten languages.

last updated 2026-07-29

Performance over time

vendor-reported independent
65752023

Baseline: No single aggregate human baseline was published; the arithmetic is grade-school level, while translation and language competence are part of the task.

Overview

MGSM (Multilingual Grade School Math) is a human-translated subset of GSM8K created to test whether chain-of-thought mathematical reasoning transfers across languages. The same 250 underlying problems appear in Bengali, Chinese, French, German, Japanese, Russian, Spanish, Swahili, Telugu, and Thai. The repository also includes English inputs and manually translated few-shot exemplars.

Because every language shares the same mathematical items, MGSM is especially useful for measuring language gaps while holding problem content fixed. It is not 2,500 independent math questions, and a macro-average can hide severe weakness in lower-resource languages.

Task details

The model receives a grade-school word problem in one target language and produces a numeric answer, often after a chain of thought. The official paper compares native-language reasoning, English reasoning, translate-to-English prompting, and multilingual exemplars. The maintained harness exposes direct and native/English-CoT task families for each language.

Example

An MGSM item is the same arithmetic story in every language, translated by human annotators rather than machine translation. A correct comparison holds the item index, decoding, and exemplars fixed and changes only the language condition; otherwise mathematical and linguistic effects become confounded.

Scoring methodology

Each language uses exact match on the extracted numeric answer. A headline MGSM score normally macro-averages language accuracies so every language receives equal weight. Report per-language values as well: two systems with the same mean may have very different equity and robustness profiles.

What it doesn’t measure

MGSM does not test advanced mathematics, proof quality, dialogue, non-numeric answers, or culturally native problem construction. Translating English-origin stories cannot fully represent how mathematical language and curricula vary across communities.

Known issues & criticisms

The sample is small and reuses public GSM8K content. Prompt language can change results substantially, and extracting Arabic numerals may advantage answers that avoid language-specific number forms. Translation quality is also an evaluation variable, not merely neutral packaging. MGSM-Pro (2026) further argues that digit-varying instantiations reveal robustness failures hidden by the original fixed numbers.

Contamination notes

Risk is high. The translations, exemplars, and targets are public, while the underlying questions inherit GSM8K’s extensive circulation. Translation reduces exact-match overlap with English but does not make the semantic tasks novel.

How to run it

Use the maintained native-CoT group:

pip install lm-eval
lm_eval --model hf --model_args pretrained=YOUR_MODEL \
  --tasks mgsm_cot_native --num_fewshot 0 --batch_size auto

For direct answering use mgsm_direct. Individual flags append language codes, such as mgsm_cot_native_sw or mgsm_direct_ja. Record the harness revision because Japanese and Chinese formatting and target extraction have received fixes.

History & lineage

MGSM was released in October 2022 as a multilingual descendant of GSM8K. Its key contribution was controlled human translation across typologically diverse languages. Later work expands the idea with more languages, locally authored data, or multiple numeric instantiations; those results are not score-compatible with the original 250-item set.

Further reading