benchwiki
reasoning

AGIEval

Saturated
2023-04-13 → today

Standardized human exams — SATs, LSATs, China's Gaokao and civil-service tests — repurposed to grade models against real human test-takers.

last updated 2026-07-25

Performance over time

vendor-reported independent
30405060708020232024human baseline 67%

Baseline: Average human test-taker across the exam sections (~67%). The paper also reports a top-percentile ceiling of ~91% (top 1% of candidates, except the lawyer qualification test's top 10%). The gauge/timeline use the average-human figure as the reference line; the 91% top-human ceiling is the harder target the paper highlights. Both are MEASURED from real exam-taker distributions, not estimated — a key differentiator of AGIEval.

Overview

AGIEval, released by a Microsoft team (Zhong et al.) in April 2023, asks a different question from most LLM benchmarks: not “can the model do a task we invented for it?” but “can the model pass the exams humans are judged by?” Its 20 tasks are drawn entirely from standardized, high-stakes human admission and qualification exams — the US SAT and LSAT, the maths problems of AQuA-RAT, and, on the Chinese side, the Gaokao (China’s national college-entrance exam) across Chinese, mathematics, physics, chemistry, biology and English, plus the civil-service and lawyer-qualification exams. The premise is that these tests were painstakingly designed by humans to discriminate human ability, so they make an unusually meaningful and human-anchored yardstick for foundation models.

That human anchoring is AGIEval’s differentiator. Because every task is a real exam, it comes with a real distribution of human scores — so the benchmark reports not an estimated or hand-waved baseline but a measured one: the average test-taker sits around 67%, and the top percentile of candidates around 91%. A model’s score can be read directly as “better or worse than the average human who sat this exam,” which is exactly the framing practitioners and the press reach for. The benchmark is also bilingual by construction — roughly half its tasks are English and half Chinese — making it one of the earliest widely-cited evals to measure high-level reasoning in both languages at once.

AGIEval is now saturated. At launch GPT-4 already crossed the average-human line on several sections (SAT Math, Gaokao English) while trailing the top-percentile ceiling; within roughly a year, frontier models pushed the 20-task aggregate past the average human and toward the 91% top-human mark, and the exam framing stopped separating strong models at the top. Its saturation carries a specific caution the human-exam framing invites (see Known issues): passing exams designed for humans is not the same as human-level general ability, and because the questions are real published exams, contamination is severe.

Task details

AGIEval is a collection of 20 exam sections, 8,062 questions total, in two answer formats:

  • Multiple-choice (18 of the 20 tasks): standard exam MCQs with a single correct option, scored by exact match on the chosen letter.
  • Cloze / fill-in (2 tasks): the Gaokao maths-cloze tasks, where the model produces a short answer (typically a number or expression) matched against the gold answer.

The English tasks are SAT (Math and Evidence-Based Reading & Writing), LSAT (analytical reasoning, logical reasoning, reading comprehension), LogiQA (English), AQuA-RAT (algebraic word problems), Gaokao-English, and the JEC-QA legal-reasoning set. The Chinese tasks are Gaokao Chinese / Mathematics / Physics / Chemistry / Biology / Math-Cloze / MathQA, the Chinese Civil Service Exam, and LogiQA (Chinese).

  • Input: an exam question (with passage/context where the exam provides one) and its answer options.
  • Output: the selected option (MCQ) or the short answer (cloze). The headline setting is zero-shot with chain-of-thought — the model reasons step by step before committing to an answer — with a few-shot and a direct (answer-only) setting also reported in the paper.
  • Aggregate: the unweighted mean accuracy across the 20 tasks (each section counts equally).

Example

From the LSAT logical-reasoning section (a task type designed to defeat shallow pattern-matching):

Question: A politician argues that because a new policy was followed by economic growth, the policy caused the growth. Which one of the following, if true, most weakens the argument?

(A) The growth began before the policy took effect. (correct) (B) The politician supported the policy publicly. (C) The policy was popular with voters. (D) Economic growth is generally desirable.

The answer requires recognizing a post hoc causal fallacy and selecting the option that breaks the causal link — the growth predating the policy. GPT-4 at launch handled many such LSAT logical-reasoning items but remained below the top-percentile human on the harder analytical-reasoning “logic games,” illustrating AGIEval’s design point: the exams already contain a difficulty gradient calibrated to separate strong human test-takers.

Scoring methodology

Accuracy by exact match on the final answer, macro-averaged across the 20 exam sections — each section counts equally regardless of question count, so the aggregate is a mean of per-exam accuracies rather than a pooled per-question accuracy. Three reporting subtleties matter:

  • Setting dependence. The paper reports zero-shot, few-shot, direct, and chain-of-thought settings, and they differ substantially on the multi-step maths and logic sections. This site’s headline axis is zero-shot CoT, the setting the paper foregrounds and the one the launch timeline uses; an AGIEval number is not comparable across settings, so always state which.
  • Two languages, one average. The aggregate blends English and Chinese exams. A single “AGIEval score” hides that a model may be much stronger in one language — the per-language and per-section breakdowns are the honest unit of comparison.
  • Human baseline is measured, not estimated. The ~67% average and ~91% top-percentile figures come from the real exam-taker distributions, which is why AGIEval’s “above/below average human” claims carry more weight than benchmarks with hand-estimated baselines. The top-percentile figure is the top 1% of candidates (top 10% for the lawyer-qualification test).

What it doesn’t measure

AGIEval measures performance on structured, single-answer academic exam questions — not open-ended reasoning, not long-horizon or multi-step agentic tasks, not tool use, not the quality of a model’s reasoning trace (only whether the final answer is correct, so a right answer from flawed working still scores). More importantly, a high AGIEval score does not establish human-level general ability: the exams test a narrow, heavily-curricularized slice of human skill under a multiple-choice format, and excelling at them is not the same as the flexible competence the “AGI” in the name evokes — a gap the benchmark’s own authors are careful about and critics have stressed. Because every question is a real published exam, a high score also increasingly reflects exposure rather than reasoning (see Contamination). It is English-and-Chinese only, and now that it is saturated it says little about frontier models.

Known issues & criticisms

  • Contamination is structural and severe. AGIEval is built from real SAT/LSAT/Gaokao/civil-service exams whose questions and worked solutions are pervasive on test-prep sites and in web-scale training corpora. There is no held-out or refreshed variant, so post-2023 scores are memorization-inclusive and should be read as an upper bound on genuine reasoning.
  • “Passing exams” ≠ human-level ability. The benchmark’s framing invites the inference that a model beating the average human test-taker has human-level intelligence. It does not: exams are a narrow, format-constrained proxy, and strong exam performance can coexist with brittle everyday reasoning. This is the most important interpretive caveat and applies to every headline “model beats humans on the SAT” claim built on AGIEval.
  • Macro-averaging distorts the aggregate. Because the 20 sections are weighted equally, a small section moves the headline as much as a large one, and the English/Chinese mix means the single number can mask large per-language gaps.
  • Saturation collapses the signal. With frontier models near the top-percentile ceiling on much of the suite, differences at the top are within noise on the aggregate; per-section scores are the only place remaining signal lives.
  • Static, ageing exam pool. The specific exam years are fixed. Unlike year-refreshed exam benchmarks, AGIEval does not rotate in new exam sittings, so it cannot separate genuine reasoning from familiarity with these particular papers.

Contamination notes

Risk: high. AGIEval’s defining strength — that its questions are real, human-calibrated exams — is also its defining contamination problem. SAT, LSAT, Gaokao and civil-service questions are among the most-reproduced content on the internet: prep books, tutoring sites, forums, and answer keys all mirror them, so the exact items and their solutions almost certainly sit in modern pretraining data. There is no canary string, no private split, and no refresh cycle; the dataset is a static snapshot of published exams. This makes AGIEval a cautionary case for the whole “give the model a human test” genre — the very fact that makes an exam a good human yardstick (it is standardized and widely studied) makes it a poor contamination-resistant model benchmark. The durable version of the idea survives in year-refreshed exam benchmarks (e.g. per-year maths-olympiad sets like AIME-by-year), which rotate in fresh questions the model cannot have seen.

How to run it

Official AGIEval repository (canonical tasks + prompts for the zero-shot / few-shot / CoT settings):

git clone https://github.com/ruixiangcui/AGIEval
# data/ holds the 20 exam sections (English + Chinese);
# the repo provides the few-shot and chain-of-thought prompt templates

OpenCompass (AGIEval is integrated as an evaluation suite):

pip install opencompass
# select the agieval datasets in your config, e.g.:
opencompass --datasets agieval --models <your_model_config>

Always report the setting (zero-shot / few-shot, direct / CoT) and, ideally, the per-language and per-section breakdown — a bare “AGIEval 72” blends two languages, twenty exams, and several prompting regimes into one uninterpretable number.

History & lineage

  • 2023-04 — AGIEval released (arXiv:2304.06364) weeks after GPT-4, framing standardized human exams as a foundation-model benchmark. Reported launch scores (zero-shot CoT, 20-task average): text-davinci-003 37.4%, ChatGPT 43.2%, GPT-4 58.4%, against a ~67% average-human and ~91% top-human baseline. GPT-4 surpassed the average human on some sections (e.g. SAT Math ~95%, Gaokao English ~92.5%) while trailing the top percentile.
  • 2023–2024 — Widely adopted as a bilingual reasoning eval, especially in Chinese-model evaluation via OpenCompass; “beats the average human on the SAT/Gaokao” becomes a common model-report line.
  • 2024 — Published as Findings of NAACL 2024. Frontier models push the aggregate past the average human toward the top-percentile ceiling; the exam framing stops discriminating at the top and AGIEval is treated as saturated.
  • 2024–2025 — The “hard human exam” torch passes to benchmarks explicitly built to resist saturation and contamination — most prominently Humanity’s Last Exam (HLE), a large expert-written exam set designed to stay hard for frontier models. AGIEval has no single drop-in descendant (its successor is null); the method survives in year-refreshed exam benchmarks that rotate fresh questions.

If you’re choosing a benchmark today: for a contamination-resistant hard-exam signal use HLE or a current frontier reasoning eval; for bilingual academic reasoning, prefer a refreshed or private-split successor. Keep AGIEval as a historical reference and a cheap regression check for smaller models.

Further reading

  • Zhong et al., 2023 — original paper: https://arxiv.org/abs/2304.06364
  • ruixiangcui/AGIEval — tasks, prompts, and per-section results: https://github.com/ruixiangcui/AGIEval
  • On the interpretive caveat, “passing exams ≠ human-level ability,” and why real-exam benchmarks contaminate: see the contamination discussion above and the year-refreshed exam approach as the durable alternative.