SuperGLUE

Overview

SuperGLUE is an eight-task English natural-language-understanding suite created as a harder successor to GLUE. By early 2019, systems had overtaken GLUE’s non-expert human estimate, so Alex Wang and colleagues retained GLUE’s broad-suite design while selecting tasks with more headroom, less training data, and more varied formats. The result spans passage question answering, natural-language inference, causal choice, multi-sentence reading comprehension, word-sense disambiguation, and pronoun resolution.

At release, the official BERT++ baseline scored 71.5 against an estimated human aggregate of 89.8. That gap closed unusually quickly: DeBERTa became the first single model reported above the aggregate human estimate in late 2020, and Vega v2 reached 91.3 on the official server in 2022. This is evidence of benchmark saturation, not proof of general human-level language understanding. The aggregate hides large differences among tasks, the human number combines different collection procedures, and several test sets are small enough that a few examples move their task scores materially.

SuperGLUE remains historically important and useful for regression testing older encoder-style systems. It is no longer a strong standalone discriminator among frontier models, particularly when evaluated through public validation splits rather than the official private-label test server.

Task details

The suite contains eight scored tasks:

The official paper reports test sizes of 3,245 BoolQ items, 250 CB items, 500 COPA items, 1,800 MultiRC answer options covering 166 questions, about 10,000 ReCoRD queries, 300 RTE items, 1,400 WiC items, and 146 WSC items. These are not one uniform unit, which is why the frontmatter leaves dataset.size null rather than presenting their sum as a meaningful suite size.

Two auxiliary diagnostics are not part of the eight-task aggregate: AX-b tests linguistic phenomena through binary entailment, and AX-g uses Winogender-style pairs to probe gender sensitivity. They are analysis sets, not extra leaderboard tasks.

Example

The original SuperGLUE paper gives this BoolQ development example. Its supplied passage says that Barq’s is bottled by Coca-Cola and describes the brand’s history.

Question: “is barq’s root beer a pepsi product”

Gold answer: No.

Example model answer: No.

This is a real benchmark item from Table 2 of the original paper, shortened here by paraphrasing the passage rather than reproducing it. A correct response requires grounding the answer in the supplied text; it should not be read as evidence that the model can answer arbitrary questions about brands.

Scoring methodology

The official SuperGLUE score gives equal weight to tasks, not examples. First calculate a score for each task. BoolQ, COPA, RTE, WiC, and WSC contribute accuracy directly. For CB, average accuracy with macro-F1. For MultiRC, average answer-level F1 with question-level exact match. For ReCoRD, average token F1 with exact match. Finally, take the arithmetic mean of those eight task scores. The AX-b and AX-g diagnostics do not enter the aggregate. This protocol is specified in Section 3.3 and Table 3 of the benchmark paper.

That design makes the headline number easy to read but not statistically uniform. A WSC accuracy computed over 146 test items receives the same aggregate weight as ReCoRD’s roughly 10,000 queries. A one-point change in any task contributes one eighth of a point overall, regardless of task size or uncertainty. CB’s macro-F1 also makes rare classes count equally, while its accuracy remains sensitive to the majority class; averaging the two is a pragmatic compromise, not a probabilistic estimator.

The 89.8 human baseline uses the same aggregate recipe, but it is not a single controlled experiment. The authors reused published human estimates for WiC, MultiRC, RTE, and ReCoRD and collected crowdworker reannotations for the other tasks. It should be treated as a reference line, not a precise ceiling. The fact that systems exceeded 89.8 while remaining below humans on individual tasks illustrates why “above human on SuperGLUE” is narrower than it sounds.

Only official test-server aggregates belong on this page’s timeline. The current lm-evaluation-harness normally evaluates labeled validation splits with its own prompts and reports an aggregate that can differ from the official submission protocol. Its README demonstrates such differences even for the same T5 checkpoint. Do not plot those validation numbers on the official leaderboard axis.

What it doesn’t measure

SuperGLUE does not measure open-ended generation quality, instruction following, factuality without a supplied context, multilingual ability, long-document understanding, tool use, or interactive problem solving. It is also not a zero-shot learning benchmark in its original form: official leaderboard systems may fine-tune separate models and use extra public or private data, subject to the benchmark rules. A high aggregate therefore shows that a training and adaptation pipeline performs well on eight fixed English classification and QA datasets, not that one model can reliably understand arbitrary language tasks.

Known issues & criticisms

• The aggregate mixes incomparable units. Accuracy, macro-F1, answer-option F1, question exact match, token F1, and entity exact match are collapsed into one arithmetic mean. Equal task weighting is explicit, but it has no statistical justification beyond simplicity; the authors say they lacked a fair alternative weighting criterion.
• Small tasks are unstable. CB, COPA, and WSC have only 250, 500, and 146 official test items. Because each task receives one eighth of the aggregate, a handful of examples can reorder tightly clustered systems. The 91.3 Vega v2 result was only 0.1 above the next entry in its archived top-ten table.
• Task-specific transfer blurs “general” understanding. The official rules do not require parameter sharing or a single shared model, and they permit auxiliary data with stated exceptions. The original BERT++ baseline itself used MultiNLI transfer for BoolQ, CB, and RTE and SWAG transfer for COPA. The suite can therefore reward eight specialized fine-tuning pipelines rather than one generally capable system.
• Labels and sources inherit dataset artifacts. CB is class-imbalanced and includes only examples above an annotator-agreement threshold; MultiRC accepts multiple answer sets; ReCoRD admits several surface forms of an entity; WSC is a binary recast assembled from small, heterogeneous sources. These choices are documented in the original paper and make errors difficult to interpret as one common failure mode.
• Private labels do not make the underlying content novel. BoolQ draws from Wikipedia, ReCoRD from CNN/Daily Mail, WiC from public lexical resources, RTE from older shared tasks, and WSC partly from published schemas and fiction. The suite predates modern foundation-model training corpora and has circulated widely. Private official labels reduce direct leaderboard tuning, but not exposure to source text or benchmark structure.
• A human aggregate is not a human ceiling. The authors combined published estimates with newly collected crowd judgments. Humans scored 100 on COPA and WSC but 80 on WiC and 51.9 exact match on MultiRC. Averaging these values conceals both annotation-protocol effects and genuine task differences.

Contamination notes

Contamination risk is high for current use. SuperGLUE is static, its training and development labels are public, its test inputs are downloadable, and both the paper and task repositories include examples. The official server originally protected test labels and limited submissions to two per day and six per month, which helped against direct test-label tuning. It could not prevent pretraining exposure to the source articles, task datasets, or benchmark discussion.

There is an additional protocol split. An official leaderboard score evaluates predictions against private test labels. A local lm-evaluation-harness run usually evaluates the labeled public validation splits, which may already appear verbatim in training data. Call the latter a “SuperGLUE validation score,” name the prompt tag and harness revision, and do not compare it directly with the 89.8 human aggregate or this page’s official-server timeline.

No refresh cycle, sealed replacement test set, or time-sliced variant exists. Decontamination by n-gram overlap can identify exact matches if the evaluator has access to a model’s training corpus, but it cannot recover the original benchmark’s lost novelty or detect paraphrased/source-document exposure.

How to run it

The maintained lm-evaluation-harness implementation exposes an eight-task tag matching its v1-style prompts:

pip install lm-eval

lm_eval --model hf \
  --model_args pretrained=google/flan-t5-base \
  --tasks super-glue-lm-eval-v1 \
  --num_fewshot 0 \
  --batch_size auto

The verified individual task flags are boolq, cb, copa, multirc, record, rte, wic, and wsc:

lm_eval --model hf \
  --model_args pretrained=google/flan-t5-base \
  --tasks boolq,cb,copa,multirc,record,rte,wic,wsc \
  --num_fewshot 0 \
  --batch_size auto

For prompts intended to reproduce the T5 paper’s text-to-text setup, use the harness’s separate tag:

lm_eval --model hf \
  --model_args pretrained=google/flan-t5-base \
  --tasks super-glue-t5-prompt \
  --num_fewshot 0 \
  --batch_size auto

The harness warns that the T5-prompt tag can fail under accelerate when ReCoRD is included. Its source README also shows that validation scores differ between the T5x and lm-eval implementations. Record the model revision, harness commit, task tag, number of shots, prompt template, and split with every result. These commands do not reproduce an official leaderboard number because they do not submit predictions to the private-label test server.

History & lineage

• 2018-04-20: GLUE was introduced as a nine-task English NLU benchmark.
• 2019-05-02: SuperGLUE was announced after systems had overtaken GLUE’s non-expert human estimate. It kept GLUE’s suite-plus-leaderboard structure but expanded beyond sentence-pair classification.
• 2019: The final suite contained eight tasks. RTE was retained directly from GLUE; WSC replaced GLUE’s problematic WNLI recast with a binary coreference formulation and non-overlapping train, development, and test sentences.
• 2020-12-29: The DeBERTa paper records the first single-model official aggregate above the 89.8 human estimate, at 89.9. This page uses that date as the saturation point.
• 2022-10-08: Vega v2 led the official leaderboard at 91.3. Its paper’s top-ten table showed several systems compressed into a 1.1-point range, further evidence that the aggregate had little frontier headroom.
• Today: SuperGLUE has no single official successor. Broad evaluations moved toward larger and more heterogeneous suites, while individual capabilities are better measured by dedicated, contamination-aware benchmarks. SuperGLUE remains useful for reproducing the encoder-transfer era and for controlled regression tests.

Further reading

• Wang et al., 2019, original SuperGLUE paper and official protocol: https://arxiv.org/abs/1905.00537
• Official tasks and dataset downloads: https://super.gluebenchmark.com/tasks
• Official diagnostic-set documentation: https://super.gluebenchmark.com/diagnostics
• Current lm-evaluation-harness task definitions and verified flags: https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/super_glue
• He et al., DeBERTa and the first single-model score above the aggregate human estimate: https://arxiv.org/abs/2006.03654
• Zhong et al., Vega v2 and an archived 2022 official-server leaderboard table: https://arxiv.org/abs/2212.01853