GPQA

Overview

GPQA — the Graduate-Level Google-Proof Q&A benchmark — is defined by a single, deliberately brutal construction step. In November 2023, Rein et al. commissioned questions from people who hold or are pursuing PhDs in biology, physics, and chemistry. Each question was then validated by two other domain experts, and — this is the innovation — handed to skilled non-expert validators who were given unrestricted access to the web and over 30 minutes per question. Those non-experts, searching Google freely, scored only 34% (and just 22% on the hardest subset). That gap is what “Google-proof” means: the questions cannot be answered by anyone, however capable, who does not already have the specialized knowledge — no amount of searching substitutes for the expertise.

That design makes GPQA a genuine expert-level yardstick, which is exactly why it became the science-reasoning benchmark of the frontier era. At launch it looked nearly unassailable: the strongest GPT-4 baseline managed about 39%, barely above the non-experts it was supposed to tower over. Then the reasoning-model era arrived and the arc turned dramatic. OpenAI’s o1 (September 2024) reached the high 70s on the headline Diamond subset; by mid-2025, frontier models were in the high 80s — crossing the line of the PhD experts who wrote and validated the questions, in those experts’ own domains. Watching a benchmark go from “barely beats a Googler” to “beats the specialists” in under two years is the story the saturation gauge on this page tells at a glance.

GPQA is now labelled nearing saturation — the wiki’s first use of that status. It is not saturated the way MMLU is: the top scores (high 80s) still sit below a ceiling that label noise puts in the low 90s, and models are only just now clearing the expert line, so the benchmark still discriminates among the best systems. But the direction is unambiguous, and one or two more model generations will likely finish it. The honest read is “excellent and still useful, but on the clock.”

Task details

Every GPQA question is 4-option multiple choice with one correct answer, scored by exact match on the chosen option. The benchmark ships as three nested subsets, and knowing which one a score refers to is essential:

GPQA main — 448 questions. The full expert-written, expert-validated set.
GPQA extended — 546 questions. The main set plus additional questions that did not pass the full validation bar; larger but noisier.
GPQA Diamond — 198 questions. The high-confidence core: questions where both expert validators answered correctly and the majority of non-experts got them wrong. This is the hardest, cleanest, and most-cited subset.

Diamond is the headline configuration — it is what essentially every leaderboard and model report means by “GPQA” — so this page treats it as primary. That is why dataset.size is set to 198 rather than 448 or 546. This reflects a general convention on the wiki: the structured size field matches the primary reported configuration, not the largest one, so the spec sheet lines up with the number readers actually encounter. When a benchmark’s headline changes subset, size follows the headline; the other subsets live here in Task details. (This convention will recur — several benchmarks report a subset by default.)

Input: a graduate-level science question and four answer options.
Output: the chosen option. In the modern convention the model reasons (chain-of-thought / reasoning tokens) before committing.
Aggregate: mean accuracy over the subset’s questions.

Example

A representative Diamond-style question (paraphrased — the real items are gated and under a no-post pledge, see Contamination):

Physics (quantum mechanics): A spin-½ particle is prepared in an eigenstate of Sₓ with eigenvalue +ℏ/2. It is then measured along an axis in the x–z plane at angle θ from the z-axis. What is the probability of obtaining +ℏ/2?

(A) cos²(θ/2) (B) sin²(θ/2) (C) ½(1 + sin θ) (correct) (D) ½(1 + cos θ)

The point of the example is not the specific answer but who can produce it: someone who knows how to project a prepared spin state onto a rotated measurement basis. A non-expert with Google can find the general Stern–Gerlach formalism but reliably fumbles the specific projection under time pressure — which is precisely the “Google-proof” gap the validation measured. A frontier reasoning model, by contrast, now works the algebra through its chain of thought and lands on (C) more often than the human experts did.

Scoring methodology

Scoring is plain exact-match accuracy on the selected option — but two reporting subtleties dominate:

Run configuration is not standardized across time. The original paper reported few-shot and chain-of-thought baselines (including a 5-shot CoT setting); the frontier era converged on zero-shot with reasoning (the model thinks before answering, no in-context exemplars). These are not the same measurement, and a “GPQA Diamond” number from 2023 and one from 2025 can differ as much from protocol as from capability. Always state shots and whether reasoning/CoT was used. This page’s headline axis is zero-shot CoT/reasoning, the dominant modern convention.
Multiple choice invites elimination. With four options, a model that cannot solve a question can still exceed 25% by eliminating obviously wrong distractors — a strategy the “Google-proof” non-expert validation did not fully control for. It means part of any score reflects test-taking rather than knowledge, and it is one reason free-response successors (which remove the options) are seen as the harder test.

What it doesn’t measure

GPQA measures expert-level factual reasoning in exactly three natural-science domains — biology, physics, chemistry — in a multiple-choice format. It is not a test of general graduate knowledge: no mathematics-proper, no engineering, no medicine, humanities, law, or social science, so “graduate-level” should be read as “graduate-level science, in three fields.” It does not measure open-ended problem solving (the options are given), long-horizon reasoning, tool use, or the correctness of the reasoning trace itself — only whether the final letter is right, so a correct answer reached by elimination or flawed working still scores. And because it is a fixed, public (if gated) set, a high score increasingly reflects exposure as well as expertise (see Contamination).

Known issues & criticisms

Tiny Diamond set → the sample-size trap. At 198 questions, GPQA Diamond has a 95% confidence interval of roughly ±5–6 points at frontier accuracy. Many head-to-head “Model A beats Model B on GPQA Diamond” claims cite gaps of 1–3 points that are pure noise. This is arguably the clearest real-world illustration of the sample-size trap in the whole wiki — see the statistical note, and distrust any small Diamond gap.
Annotation errors cap the ceiling below 100%. Experts disagreed on a nontrivial fraction of questions; the paper reports that discounting mistakes the experts themselves later acknowledged raises expert accuracy from 65% to ~74% on the main set, implying a few percent of items are mislabeled or contestable. The practical ceiling is therefore in the low 90s, not 100 — which is why frontier scores in the high 80s already signal near-saturation.
“Beats PhD experts” is baseline-dependent. The paper’s Diamond expert accuracy is 81.3%; OpenAI’s separately recruited experts scored 69.7% on the same subset. A model at, say, 78% “beats experts” against one baseline and not the other. Claims of super-expert performance should always name the expert pool.
Multiple-choice elimination. Four options let models bank partial credit via distractor elimination, inflating scores relative to a free-response version of the same questions (see Scoring methodology).
Only three domains. GPQA’s “graduate-level” brand oversells its scope: it is silent on most of what a graduate education covers.

Contamination notes

Risk: medium, and trending upward. GPQA is newer and far smaller than MMLU, and it ships with a real mitigation MMLU never had: the HuggingFace dataset is gated behind an access agreement, and users are asked not to post questions or answers online “to reduce the risk of leakage into foundation model training corpora.” That is a genuine effort — but it is a voluntary pledge, not a technical control, and it belongs to the same failed category as BIG-Bench’s canary string: opt-in decontamination that depends on everyone’s good behavior. In practice, Diamond questions have circulated in discussions, papers, and reproductions since 2023, and the set is fully static with no refreshed variant. So the honest label is medium now, drifting toward high as the questions age into more training corpora. Treat post-2024 Diamond scores as containing some memorization, and weight the trajectory (still climbing, still discriminating) over any single absolute number.

How to run it

Inspect (AISI’s harness; gpqa_diamond is the standard task):

pip install inspect-ai
inspect eval inspect_evals/gpqa_diamond --model openai/gpt-4o
# zero-shot chain-of-thought over the 198 Diamond questions

lm-evaluation-harness:

pip install lm-eval
lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-3.1-8B \
  --tasks gpqa_diamond_zeroshot \
  --batch_size auto
# gpqa_main_zeroshot and gpqa_extended_zeroshot expose the larger subsets

Official dataset (gated — you must accept the agreement and the no-post pledge):

# https://huggingface.co/datasets/Idavidrein/gpqa  → request access, then:
from datasets import load_dataset
ds = load_dataset("Idavidrein/gpqa", "gpqa_diamond")

Always report the subset (Diamond / main / extended), the shots and reasoning setting, and — given the ±6-point Diamond CI — ideally a confidence interval. A bare “GPQA 84” is three different benchmarks and several protocols rolled into one uninterpretable number.

History & lineage

2023-11 — GPQA released (arXiv:2311.12022; later a COLM 2024 spotlight). PhD-written, expert-validated, and Google-proofed against non-experts (34% with web access; 22% on Diamond). The strongest GPT-4 baseline reaches ~39% on the main set — barely above the non-experts — establishing GPQA as the hardest science benchmark of its moment.
2024-05 → 2024-09 — GPT-4o reaches ~56% on Diamond; then OpenAI’s o1 posts ~78%, the reasoning-model inflection point. OpenAI reports its recruited PhD experts at 69.7% on Diamond, prompting the “o1 beats PhD experts” framing — true against that pool, short of the paper’s 81.3%.
2025 — Frontier reasoning models (e.g. Grok 4, ~87%) cross the paper’s expert line in the models’ own domains, with the label-noise ceiling in the low 90s now the binding constraint.
2024–2026 — As GPQA nears saturation, the “hard expert exam” role passes to broader, harder, contamination-resistant successors — most prominently Humanity’s Last Exam (HLE), a large expert-written multi-domain set built to stay hard for frontier models. (Its page isn’t written here yet, so this page’s successor slug does not yet resolve to a link.)

If you’re choosing a benchmark today: GPQA Diamond remains a strong, discriminating science-reasoning signal — but respect its ±6-point noise floor, and reach for HLE or a free-response successor when you need a harder or larger-sample test.

shots	zero-shot
reasoning	cot
temperature	—
pass@k	pass@1
tools	no
harnesses	GPQA (official) Inspect `gpqa_diamond` lm-evaluation-harness `gpqa_diamond_zeroshot`

Performance over time