benchwiki
maths

TabMWP

Saturated
2022-09-29 → today

Grade-level math word problems that require combining a question with structured, textual, or image-rendered tables.

last updated 2026-07-29

Performance over time

vendor-reported independent
6070809010020222023human baseline 90.22%

Baseline: Official test-set estimate reported by the benchmark authors; subgroup accuracy varies by free-text, multiple-choice, and answer type.

Overview

TabMWP (Tabular Math Word Problems) tests grade-level mathematical reasoning over a question plus a table. Its 38,431 problems were collected from open educational websites and expose each table as an image, semi-structured text, and structured data. That makes it possible to separate table perception from reasoning.

The benchmark was introduced with PromptPG, which learned to select in-context examples for GPT-3. It subsequently became saturated under the easier-to-parse Text-GT protocol: the official leaderboard reports tool-augmented GPT-4 systems above the benchmark’s human estimate. Image-table evaluation remains a distinct multimodal diagnostic and must not be conflated with Text-GT scores.

Task details

TabMWP contains 23,059 training, 7,686 development, and 7,686 test questions. Problems may be free-response or multiple choice, and answers may be integers, decimals, extracted strings, booleans, or other text. Every item includes a gold solution explaining the multi-step reasoning.

Example

A typical item supplies a small table of quantities or prices and asks the model to combine selected cells using a stated arithmetic relation. The model must identify the relevant row and column before calculating. The official project offers an interactive explorer rather than requiring examples to be copied into secondary documentation.

Scoring methodology

The official metric is exact answer accuracy, with normalization appropriate to the answer type. The leaderboard breaks results down by free-text versus multiple choice and by integer, decimal, extractive, boolean, and other answers. Table format is part of the protocol: a score using gold text does not measure OCR or visual table understanding.

What it doesn’t measure

TabMWP does not target advanced mathematics, proof writing, long documents, or arbitrary real-world spreadsheets. Most questions use grade-level arithmetic over small tables, so high accuracy does not establish robust data-analysis ability.

Known issues & criticisms

The official leaderboard mixes architectures and adaptation regimes unless readers inspect its Table, Method, and Type columns. Fine-tuned systems, few-shot language models, code generators, and tool agents solve meaningfully different tasks. The static public answers also make current contamination difficult to exclude.

Contamination notes

Risk is high. The complete dataset, including test labels and worked solutions, has been public since 2022. Educational source material may also occur independently in pretraining corpora. Use a held-out private table collection for capability claims.

How to run it

Clone the official repository and follow its environment instructions:

git clone https://github.com/lupantech/PromptPG.git
cd PromptPG
pip install -r requirements.txt
python evaluate_acc.py --help

The exact inference entry point depends on run_gpt3, run_t5, or run_tapex. Report the split, table representation, prompt examples, answer parser, tools, and repository commit.

History & lineage

TabMWP was released on 2022-09-29 and published at ICLR 2023. It is not a descendant of GSM8K: its defining addition is semi-structured table context. Later table-reasoning systems retained it as a component benchmark, but no official refreshed successor exists.

Further reading