benchwiki
knowledge

SimpleQA

Active
2024-11-07 → today

4,326 short fact-seeking questions chosen to stump GPT-4 — a hallucination test graded correct / incorrect / not-attempted.

last updated 2026-07-21

Performance over time

vendor-reported independent
15253545556520242025

Baseline: No human baseline — and arguably none is meaningful. The questions were deliberately selected to be obscure (adversarially filtered so frontier models got them wrong), so a human sitting the test closed-book would mostly measure luck, and open-book would measure Googling, not knowledge. The paper reports a 94.4% agreement rate from a third annotator, but that measures dataset quality (are the answers indisputable?), not human question-answering ability. This is the first benchmark on the wiki where a human reference line is not just missing but conceptually beside the point.

Overview

SimpleQA is the benchmark that put a number on hallucination. Released by OpenAI (Wei et al.) in November 2024, it asks 4,326 short, fact-seeking questions, each with a single indisputable answer — “Who received the IEEE Frank Rosenblatt Award in 2010?”, “What day, month, and year was X born?” The questions were adversarially collected against GPT-4: a question was kept only if frontier models of the day got it wrong, so the set is, by construction, the stuff models don’t reliably know. With browsing turned off, a SimpleQA score measures parametric knowledge — what the model has actually stored versus what it confidently makes up.

This is the precise complement to MMLU. MMLU asks a model to recognize the right answer among four options; SimpleQA asks it to recall the answer with nothing to choose from. The MMLU page sets up exactly this gap — recognition inflates scores because elimination and partial cues do work the model can’t do unaided. SimpleQA is the benchmark that quantifies that gap: the same model that scores ~88% recognizing MMLU answers scores under 40% recalling SimpleQA facts. Multiple choice measured a ceiling; open-ended recall reopened the floor.

But SimpleQA’s real design insight is in its grading, which makes it two benchmarks in one — a factuality test and a calibration test. Every answer is classified into three outcomes, not two: correct, incorrect, or not attempted (the model declined or hedged). That third category is the whole point. A model that says “I don’t know” when it doesn’t know is behaving well; a model that confidently states a wrong birthdate is hallucinating. By separating humility from ignorance, SimpleQA rewards a model for knowing its own limits — the paper’s stated ideal is “get as many questions correct as possible while not attempting the ones you’re not confident about.” That dual identity — factuality + calibration — is what distinguishes it from a plain accuracy benchmark.

SimpleQA is active — the first benchmark on this wiki to hold that status, and it earns it. Frontier models climbed from GPT-4o’s 38% correct at launch to GPT-4.5’s ~62% by early 2025, but the ceiling is nowhere in sight, and reasoning tuning doesn’t automatically help (o1-mini managed just 8% correct while declining 29% of questions). The score is still rising toward an unknown top — which is exactly what a useful, unsaturated benchmark looks like.

Task details

Each item is a single question with a short, unambiguous, verifiable answer (a name, date, number, or place). The design criteria the authors enforced: the answer must be indisputable and stable, gradable by simple string/semantic comparison, and — critically — hard, meaning at least one of four frontier-model completions was wrong at collection time.

  • Input: one fact-seeking question. No options, no context, no browsing — the model answers from its parameters.
  • Output: a short free-text answer, which the model may also decline (“I don’t know”).
  • Grading: a prompted LLM classifier compares the answer to the reference and returns correct (the answer contains the reference without contradiction), incorrect (it contradicts the reference), or not attempted (no answer / hedge, no contradiction).
  • Aggregate: the three counts roll up into the metrics below.

dataset.size is 4,326, the full released set — the primary configuration, per the wiki’s size-matches-primary convention (see GPQA).

Tools-off is constitutive, not a setting. The entire premise is closed-book recall. Turn browsing on and SimpleQA stops measuring knowledge and starts measuring search — a different, much easier task. Some labs report browse-enabled SimpleQA numbers, and they are not comparable to the closed-book scores here; treat a “SimpleQA with tools” figure as a separate benchmark and never place it on the same axis.

Example

Question: Who received the IEEE Frank Rosenblatt Award in 2010?

Correct: “Michio Sugeno.” — contains the reference answer, no contradiction → correct. Hallucinated: “The 2010 award went to Geoffrey Hinton.” — a confident, specific, wrong answer → incorrect. Humble: “I’m not certain who won that specific year.” — declines without asserting a falsehood → not attempted.

The three responses are the benchmark in miniature. Binary accuracy would lump the humble answer in with the hallucination (both “not correct”); SimpleQA separates them, and that separation is what lets it score calibration. A model that produced the third answer instead of the second is strictly better behaved even though neither scored a point.

Scoring methodology

Grading is LLM-as-judge — the wiki’s first benchmark where the primary metric is judge-scored (TruthfulQA used a judge only for its generation variant). The judge is a prompted ChatGPT classifier; OpenAI’s reference implementation uses GPT-4o. From the three-way outcome come three numbers:

  • correct (the headline, metric.primary): fraction of all questions answered correctly. This is the most-quoted SimpleQA number and the one in model cards.
  • correct given attempted: fraction of attempted questions answered correctly — accuracy when the model chooses to answer. It ignores declining, so a cautious model can score high here while its “correct” stays low.
  • F-score: the harmonic mean of the two, 2c/(2c+2i+n). It is high only when a model both knows a lot and declines when unsure — the single summary of the factuality-plus-calibration ideal, and the metric SimpleQA Verified adopts as F1.

Why the not-attempted category is the design’s core: it turns SimpleQA into a calibration benchmark. Two models with identical knowledge can score very differently — the one that guesses on everything converts its uncertainty into hallucinations (high incorrect), while the one that declines when unsure converts it into abstentions (high not-attempted, higher correct-given-attempted, better F). SimpleQA can therefore reward a model for epistemic humility, something a binary right/wrong benchmark structurally cannot. Reading only “correct” throws that away; always read it beside “not attempted.”

What it doesn’t measure

SimpleQA measures short-form, closed-book, single-fact recall in English — and nothing beyond it. It says nothing about long-form factuality: a model can nail isolated facts yet hallucinate fluently across a paragraph, which is why long-form factuality evaluations (e.g. FActScore-style and LongFact work) exist as a separate line. It does not measure reasoning, synthesis, or usefulness — only whether one atomic fact is right. Because tools are off by design, it does not reflect a retrieval-augmented deployment where the model can look things up (arguably the more common real-world setting). And its adversarial, obscure question distribution is deliberately unrepresentative of what users typically ask, so a SimpleQA percentage is a stress-test of parametric knowledge, not an estimate of everyday factual reliability.

Known issues & criticisms

  • LLM-grader error and version drift. Scores depend on a judge model that itself makes mistakes and changes over time. Two “SimpleQA 55” numbers graded by different judge models (or prompt versions) are not strictly comparable — the same lesson TruthfulQA learned when its fine-tuned GPT-judge was deprecated, recurring in modern form. Report the grader.
  • Answer-key staleness. “Single indisputable answer” decays: office-holders change, records are broken, and some facts are more contested than the annotators judged. A static 2024 answer key slowly accumulates entries that are no longer (or were never quite) indisputable.
  • Adversarial-against-GPT-4 selection bias. Questions were kept because 2024 OpenAI models failed them, so difficulty is calibrated to one model family’s blind spots. This can subtly advantage or disadvantage other model families on the exact axis they differ from GPT-4, a bias other researchers have noted and one motivation for SimpleQA Verified’s re-collection.
  • Memorization of a hallucination benchmark. Because it is public and static, models can improve by having ingested the discussed Q&A pairs rather than by knowing more — the contamination irony (see below).
  • Short-form only. A high score does not imply low hallucination in long-form generation, the setting users actually encounter.

Contamination notes

Risk: medium, with a distinctive twist. SimpleQA is public, ungated, and fully static, so the ordinary leakage clock is running: the 4,326 Q&A pairs are now spread across papers, blog posts, and leaderboard write-ups, and post-2024 training corpora increasingly contain them. The twist is the irony — a benchmark that measures hallucination can be defeated by memorization. A model that has seen the answer key scores higher on “correct” without having become any less prone to making things up, which is precisely the failure the benchmark exists to catch. There is no refresh cycle to counter this. The main mitigation is external: SimpleQA Verified (Google DeepMind, 2025) re-collects and cleans a 1,000-question subset partly to reduce leakage and label noise, and Chinese SimpleQA extends the method to a fresh language. Treat rising SimpleQA scores on very recent models as part-knowledge, part-exposure, and prefer the freshest variant when contamination matters.

How to run it

simple-evals (OpenAI’s official harness; grader required):

pip install simple-evals   # or clone github.com/openai/simple-evals
# SimpleQA needs a grader model; set your OpenAI key so the ChatGPT classifier can run
python -m simple_evals.simpleqa --model <your-model> --grader gpt-4o

Inspect (AISI harness):

pip install inspect-ai
inspect eval inspect_evals/simpleqa --model openai/gpt-4o
# keep tools OFF — browse-enabled runs measure search, not knowledge

Always report the grader model, that tools were off, and all three numbers (or at least correct and not attempted). A bare “SimpleQA 50” hides whether the model is knowledgeable or merely bold, and whether a different judge would agree.

History & lineage

  • 2024-11 — OpenAI releases SimpleQA (arXiv:2411.04368), reframing factuality as short-form parametric recall with a three-way (correct / incorrect / not-attempted) grade. Launch scores are low: GPT-4o 38.2% correct, o1-preview 42.7%, and cautious models declining a third of questions.
  • 2024-11 — Chinese SimpleQA (Alibaba, arXiv:2411.07140) ports the methodology to Chinese: 3,000 questions across 6 topics, same three-way grading.
  • 2025-02 — GPT-4.5 reaches ~62.5% correct, roughly halving GPT-4o’s hallucination rate on the benchmark — evidence the test is climbing but far from solved.
  • 2025-09 — SimpleQA Verified (Google DeepMind, arXiv:2509.07968) re-collects and cleans a 1,000-prompt subset (de-duplicated, topic-balanced, source-reconciled, improved autorater) to fix SimpleQA’s noisy labels and topical bias; it reports F1 and becomes the more reliable variant (Gemini 2.5 Pro leads early results).

The lineage is a benchmark improving in public: the OpenAI original defined the task; Chinese SimpleQA generalized the language; SimpleQA Verified hardened the labels. Neither replacement supersedes the original as the named benchmark yet, so this page’s successor is null — use SimpleQA Verified when you need clean labels, and the original for continuity with 2024–25 model cards.

Further reading