benchwiki
knowledge

Humanity's Last Exam

Active
2025-01-24 → today

~2,500 expert-written questions across 100+ subjects, each filtered to stump frontier models — designed to be the final closed-ended academic benchmark.

last updated 2026-07-21

Performance over time

vendor-reported independent
0102030402025

Baseline: No human baseline — and here the concept itself breaks down. HLE spans 100+ expert subjects from classics to rocket engineering; no single human could sit the whole exam, so an aggregate 'human score' is undefined in principle, not merely unmeasured. This differs from SimpleQA, where a human COULD take the test but the score would mostly measure Googling: HLE is superhuman in BREADTH by construction. The only meaningful human reference is per-question — each was solved by the domain expert who wrote it — which is exactly why the benchmark exists: to test whether one model can match the union of many human specialists.

Overview

Humanity’s Last Exam is a benchmark with a thesis baked into its name. Released in January 2025 by the Center for AI Safety and Scale AI, it was built to be the final closed-ended academic benchmark — the argument being that if models saturate a broad, expert-level, unambiguous-answer exam this hard, then closed-ended question answering has nothing left to measure, and evaluation must move to open-ended, agentic, and interactive tasks. The name is not marketing; it is a claim about where benchmarking runs out of road.

Its construction is GPQA’s method scaled up on every axis. GPQA had domain experts write graduate-level science questions and validated them as “Google-proof” against skilled non-experts. HLE takes the same expert-authored, failure-filtered recipe and enlarges it enormously: nearly 1,000 subject-matter experts across 500+ institutions in 50 countries wrote ~2,500 questions spanning 100+ subjects — not just physics, chemistry, and biology, but classics, ecology, linguistics, and rocket engineering. Contributors were paid from a prize pool for accepted questions, and every question had to pass a hard filter — frontier models had to fail it at submission time — followed by expert peer review. Where GPQA is deep in three sciences, HLE is deep across the whole map of human expertise. It is, in effect, GPQA’s difficulty and provenance discipline generalized to “everything a specialist knows.”

Two design choices make it the most rigorously-guarded benchmark catalogued here. First, a private held-out question set: alongside the public questions, CAIS/Scale keep a private set and compare a model’s public-vs-private performance to detect benchmark-specific overfitting — the wiki’s first partially-private design (see Contamination). Second, it reports calibration, not just accuracy: launch models were not only wrong on ~90% of questions but confidently wrong, stating ~90%+ confidence while failing. That “confidently wrong on expert questions” finding is a core teaching point — it is the same overconfidence problem SimpleQA attacks with its not-attempted option, approached from the other side (measuring the miscalibration directly rather than rewarding abstention).

HLE is firmly active — the clearest active case on the wiki. Launch scores were single digits; eighteen months later the no-tools frontier is around 38% (Gemini 3 Pro). That is a steep climb, but the exam is nowhere near solved, and the “last exam” framing means its eventual saturation would be a milestone for the whole field, not just a retired benchmark.

Task details

HLE is ~2,500 questions across 100+ subjects, in two formats:

  • Exact-match short answer (~80%): the model must produce a specific answer (a value, name, expression, or phrase) that is checked for equivalence against a known, unambiguous solution.
  • Multiple choice (~20%): used where a short answer would be ambiguous; the model selects the correct option.

~10% of questions are multimodal — they require reading an image (a diagram, chemical structure, figure, or plot) alongside the text. This is a minority of the benchmark, and HLE’s leaderboard identity is a knowledge/reasoning exam rather than a vision benchmark, which is why this page files it under knowledge and flags the multimodal share here rather than as a primary capability. Comparability trap: a model without vision (or evaluated text-only) can only be scored on the ~90% text subset, so a “text-only HLE” number is not comparable to a full-benchmark number — some launch results (e.g. DeepSeek-R1) were text-only for exactly this reason. Always state whether the multimodal questions were included.

dataset.size is 2,500, the current public set, following the size-matches-primary convention (see GPQA). The paper originally described 3,000 questions; post-launch expert review removed flawed items, refining the public set to ~2,500.

Tools off is the canonical configuration. The headline HLE number is closed-book — no browsing, no code execution, no retrieval. This matters enormously (see Timeline): the same model scores far higher when allowed to search.

Example

HLE questions are, by design, ones a frontier model got wrong. A representative short-answer item (paraphrased — the set is gated and under a no-redistribution pledge):

Classics / philology: In a specified line of a named Greek tragedy, identify the metrical foot at a given position and the technical name for the resolution occurring there.

Reference answer: a specific, checkable term (e.g. “anapaest; resolution of the longum”). The model’s free-text answer is compared for equivalence by the GPT-4o judge.

The point is breadth-times-depth: no single human holds the specialist knowledge to answer both this and a rocket-propulsion question and an organic-synthesis question, yet HLE asks one model to do all of it. A model that scores 38% is not 38% of the way to “a smart generalist human” — there is no such human — it is matching expert answers across a union of fields no individual commands.

Scoring methodology

The headline metric is accuracy (no tools), and grading is LLM-as-judge for the 80% short-answer portion: a pinned GPT-4o (gpt-4o-2024-08-06) with structured decoding checks whether the model’s answer is equivalent to the reference. This inherits the judge-provenance caveats established on the SimpleQA page — an imperfect judge that can drift across versions, so two HLE numbers graded by different judges are not strictly comparable; pin the grader.

The second headline number is calibration. Each answer carries the model’s stated confidence, and HLE reports an RMS calibration error — the root-mean-square gap between confidence and correctness. The launch finding was stark: models posted ~80–93% calibration error, i.e. they were nearly always confident and nearly always wrong. This is the direct-measurement counterpart to SimpleQA’s design: SimpleQA rewards a model for declining when unsure (its not-attempted category), while HLE simply measures how badly confidence and correctness diverge. Two attacks on the same overconfidence problem — read HLE’s accuracy and calibration error together, because a model can raise one while the other stays bad.

What it doesn’t measure

HLE measures closed-ended, single-answer academic knowledge and reasoning — and by its own thesis, that is the thing it intends to be the last word on. It does not measure open-ended generation, long-horizon planning, tool use, or agentic problem-solving; indeed, the whole point is that once closed-ended testing saturates, those are what remain. It is not a test of everyday usefulness — its questions are deliberately at the edge of expert knowledge, unrepresentative of normal use. The no-tools headline says nothing about retrieval-augmented deployment (where models can look things up), and the tool-augmented scores are a different measurement entirely. It is English-only, and its ~10% multimodal share makes it a light vision test at best.

Known issues & criticisms

  • Answer-key disputes. On a fraction of questions, domain experts have contested the gold answers post-launch; some items were removed in the 3,000→2,500 refinement. As with any expert benchmark at the edge of knowledge, a small residual error rate caps the effective ceiling below 100%.
  • LLM-judge grading errors. The GPT-4o equivalence check can mis-grade correct-but-unusually-phrased answers or accept near-misses; the judge is a source of noise and of cross-version incomparability.
  • Failure-filter selection bias, magnified. Questions were kept only if 2024-frontier models failed them, so difficulty is calibrated to that generation’s blind spots — the same adversarial-selection bias noted on SimpleQA and GPQA, amplified by HLE’s scale and by using several frontier models as the filter.
  • Multiple-choice guessable floor. The 20% MC portion has a nonzero guess rate, which slightly inflates very low scores — relevant when comparing single-digit launch numbers.
  • Scores go stale fast. The frontier climbs monthly; any published HLE number is dated almost immediately. The wiki’s mitigation is discipline: every figure here is dated and last_updated is bumped on each revision. Treat an undated HLE score as unusable.
  • Configuration confusion. No-tools, tool-augmented, and multi-agent “heavy” scores are routinely conflated in reporting, making a model look far better or worse than a like-for-like comparison would (see Timeline).

Contamination notes

Risk: medium, and it earns the “medium” through the strongest mitigation design catalogued here rather than through obscurity. The public ~2,500-question set is static, gated on HuggingFace (users accept an agreement and are asked not to re-share or re-upload), and — inevitably — discussed online, so pure leakage risk is real and rising. What holds the risk at medium is the private held-out set: CAIS/Scale can compare a model’s public-set and private-set accuracy, and a large gap flags benchmark-specific overfitting or contamination that a fully public benchmark could never detect. This is static-but-audited, a genuine advance over the “please don’t train on this” honor systems that failed for BIG-Bench and the pure-public sets of MMLU-era benchmarks. It is not immunity — the public questions can still leak, and the failure-filter bias remains — but the audit catches the crudest gaming, and the design is the current best practice for a static benchmark.

How to run it

Official HLE harness (grader required; keep tools OFF for the canonical number):

git clone https://github.com/centerforaisafety/hle
# request access to the dataset: https://huggingface.co/datasets/cais/hle
# evaluation needs a GPT-4o judge key for the short-answer equivalence check
python -m hle.run --model <your-model> --judge gpt-4o-2024-08-06 --tools none

Inspect (AISI harness):

pip install inspect-ai
inspect eval inspect_evals/hle --model openai/gpt-5
# report the tool setting explicitly; a tools-on run is a different benchmark

Always report: the tool configuration (no-tools / tools / agent), whether multimodal questions were included, the judge model version, and the date. A bare “HLE 41” is uninterpretable without all four.

History & lineage

  • 2023–2024 — MMLU saturates; GPQA raises the bar with expert-written, Google-proof science questions but is limited to three domains and, by 2025, is nearing saturation itself. The field needs a broader, harder closed-ended exam.
  • 2025-01 — HLE launches (arXiv:2501.14249): ~1,000 experts, 100+ subjects, a hard failure-filter, a private held-out audit, and a calibration metric. Launch scores are single digits with ~90% calibration error — models confidently wrong across the board.
  • 2025 — The no-tools frontier climbs steeply: Grok 4 ~24% (Jul), GPT-5 ~25% (Aug), Gemini 3 Pro ~38% (Nov). Tool-augmented and agentic systems (Deep-Research-style browsing, multi-agent “heavy” modes) post substantially higher numbers, splitting HLE reporting into incomparable configurations.
  • 2026 — Still active and still climbing; the open question is whether “the last exam” saturates before the field’s attention shifts wholesale to agentic and open-ended evaluation.

HLE is the designated successor to GPQA — GPQA’s page links here as its successor. It has no successor of its own yet (and by its thesis, its successor would not be another Q&A benchmark but a move to open-ended/agentic testing), so this page’s successor is null.

Further reading