benchwiki
multimodal

MathVista

Nearing saturation
2023-10-03 → today

Visual mathematical reasoning across diagrams, plots, tables, scientific figures, puzzles, and natural images.

last updated 2026-09-05

Performance over time

vendor-reported independent
405060708020232024human baseline 60.3%

Baseline: Amazon Mechanical Turk average on testmini; annotators had a high-school diploma or higher. This is not an expert-mathematician ceiling.

Overview

MathVista evaluates mathematical reasoning that cannot be reduced to plain text alone. Its 6,141 questions span geometry diagrams, function and statistical plots, tables, scientific figures, puzzle patterns, synthetic scenes, and natural images. It combines 28 existing math/VQA datasets with three newly created sources: IQTest, FunctionQA, and PaperQA.

The benchmark helped expose a useful distinction between language-only calculation and reasoning grounded in a visual representation. It is now a strong diagnostic suite rather than a pristine frontier separator: the widely used testmini split is public, and the official leaderboard reported several systems above its 60.3% crowd-worker baseline by late 2024.

Task details

Items are either multiple choice or free response. Free-response answers are short integers, floating-point values, or lists. The model receives an image plus a question and may also receive answer choices and a task-specific formatting instruction.

MathVista has a public 1,000-example testmini split and a 5,141-example test split. The latter’s labels were withheld at release for server-side evaluation.

Example

A geometry item shows a labeled diagram and asks for a missing angle, with four numeric options. A correct response identifies the relevant geometric relation and ends with the selected option letter. Merely reading visible text is insufficient when the diagram encodes the required relationships.

Scoring methodology

The original pipeline has three stages: generate a response, extract a short answer, then normalize and compare it with the reference. The paper used GPT-4 as an answer extractor and reported more than 99.5% extraction accuracy on a 200-example study. Multiple-choice answers become option letters; numeric free-form answers are normalized before exact comparison.

That makes the headline accuracy partly dependent on extraction. Modern reimplementations may use different parsers, so comparisons should disclose the evaluator version. Never mix testmini and full test scores.

What it doesn’t measure

MathVista does not isolate pure visual perception from mathematical reasoning, and its aggregate mixes school levels, reasoning types, and source datasets. It does not evaluate formal proofs, open-ended scientific discovery, video, audio, or interactive tool use.

Known issues & criticisms

  • Twenty-eight component datasets predate MathVista, so memorization or familiarity with source distributions can help.
  • The crowd-worker baseline is not an expert ceiling; exceeding 60.3% does not mean superhuman visual mathematics.
  • An LLM-based answer extractor adds a model-dependent layer between raw response and score.
  • The mixture of multiple-choice and free-form questions gives different opportunities for guessing and parser failure.
  • Research on multimodal MCQ reliability, including MMEvalPro, shows that questions inherited from suites including MathVista can sometimes admit text-only shortcuts unless perception is explicitly checked.

Contamination notes

The official full-test labels were hidden at launch, but the source material and testmini labels are public. Because 28 component datasets are older public benchmarks, contemporary models may have seen questions, images, or close variants. Use the hidden-label test when available and pair MathVista with newer private or refreshed visual-math sets.

How to run it

git clone https://github.com/lupantech/MathVista.git
cd MathVista
pip install -r requirements.txt
python evaluation/evaluate.py --answer_file outputs.json --score_file scores.json

Consult the repository for the current response-generation adapters and exact file arguments. State the split, prompt style, reasoning/tool setting, and answer extractor in every result.

History & lineage

MathVista was submitted on 2023-10-03 and appeared at ICLR 2024. The original paper reported GPT-4V at 49.9% versus a 60.3% human baseline on testmini. The official project later recorded GPT-4o at 63.8% and o1 at 73.9%, alongside multiple models exceeding the average human result. No single official successor is recorded here.

Further reading