GLUE

Overview

GLUE, the General Language Understanding Evaluation, combined nine established English natural-language-understanding datasets into a common platform and a single headline score. Its tasks span grammatical acceptability, sentiment, paraphrase and duplicate detection, semantic similarity, natural language inference, question answering recast as inference, and Winograd-style coreference. A separate hand-built diagnostic set probes linguistic phenomena in an NLI format.

GLUE became the scoreboard for the first wave of pretrained encoders. BERT raised the official score to 80.5, and within a year RoBERTa reported 88.5, above the 87.1 conservative non-expert human estimate. This is why GLUE is marked saturated, not because every constituent task is solved, but because the aggregate stopped providing useful headroom and can conceal regressions behind ceiling-level tasks. The authors introduced SuperGLUE specifically as a harder successor.

Task details

GLUE is a suite, not one task:

Models are conventionally pretrained once and then fine-tuned separately for every task. They submit predictions for the nine primary tasks plus the auxiliary diagnostic set. The official FAQ distinguishes the nine primary submissions from the diagnostic file; the diagnostic analysis does not become a tenth term in GLUE Score.

Example

A real CoLA training example from the officially maintained dataset card:

Sentence: Our friends won’t buy this analysis, let alone the next one we propose.

Gold label: acceptable

A GLUE system fine-tuned for CoLA emits the acceptable class. Another task in the same suite may instead receive a sentence pair and emit entailment, a paraphrase label, or a real-valued similarity score. There is no universal GLUE prompt or output format.

Scoring methodology

The official GLUE Score is a macro-average over nine task-level values. Paired metrics are first combined within a task: MRPC and QQP combine accuracy with F1; STS-B combines Pearson with Spearman correlation; MNLI combines matched and mismatched accuracy. Those task values then receive equal weight even though their training sets differ by orders of magnitude.

Important protocol distinctions:

• Official test versus local validation. The leaderboard evaluates submitted predictions against hidden labels. lm-evaluation-harness states that it defaults to validation sets because GLUE test labels are not publicly accessible.
• Nine tasks versus eight. Many later papers omit problematic WNLI and report an eight-task development average. For example, DeBERTaV3’s 91.37 is explicitly an eight-task GLUE average; it is not comparable to the official nine-task timeline above.
• Metric selection. Papers sometimes report only F1 for MRPC/QQP or only Spearman for STS-B instead of the official paired-metric contribution.
• Task reformulations. The RoBERTa paper documents a pairwise-ranking QNLI submission and a reformatted WNLI procedure. Those choices can improve the leaderboard score without being the same classification protocol used in earlier work.
• Ensembling and transfer. RoBERTa’s 88.5 test entry ensembles 5-7 models per task and initializes several tasks from an MNLI-fine-tuned model. It should not be compared as though it were a single checkpoint under a uniform recipe.

What it doesn’t measure

GLUE does not directly measure generation quality, instruction following, factual reliability, long context, multilingual ability, safety, tool use, or interactive reasoning. Even within NLU, its aggregate cannot identify why a model improved: a point gained on a large sentiment task and a point gained on low-resource entailment receive the same weight. The static English datasets also capture the genres and annotation conventions of their source corpora rather than general language understanding in the broad sense.

Known issues & criticisms

• Aggregate saturation. The SuperGLUE paper states that GLUE performance had surpassed non-expert human performance and offered limited headroom only about a year after launch. Saturation of the average does not imply that every underlying task is reliable or solved.
• NLI annotation artifacts. Gururangan et al. show that hypothesis-only classifiers reach 53% on MultiNLI because crowd-writing patterns correlate with labels. MNLI and tasks derived from entailment can therefore reward shortcuts.
• Syntactic heuristics. McCoy et al. find that BERT models trained on MNLI perform poorly on HANS examples that break lexical-overlap, subsequence, and constituent heuristics, despite strong in-distribution GLUE results.
• WNLI is structurally problematic. The GLUE dataset card documents shared hypotheses across training and development with flipped labels, making the development split adversarial to memorization. Many papers omit it, use majority prediction, or reformulate it, changing the aggregate.
• Small-task instability. RTE, MRPC, CoLA, and STS-B have small training sets. Fine-tuning seed selection and hyperparameter sweeps can move results materially; reporting only the best seed overstates reliability.

Contamination notes

Risk is high for modern pretrained models even though the official labels are not uniformly public. GLUE is static and assembled from older, widely distributed datasets. Training and development splits are public, Hugging Face mirrors all configurations, and several constituent test sets or source corpora predate GLUE. Private labels prevent direct label lookup for part of the official test, but they do not prevent a model from seeing the input text or closely related source material during pretraining.

The original GLUE paper notes that four tasks used privately held test data. That was a meaningful submission safeguard in 2018, not a refresh mechanism. For current models, report any corpus overlap filtering task by task; a single suite-level contamination claim is too coarse.

How to run it

For the historical supervised workflow, download all tasks with the official baseline repository:

git clone https://github.com/nyu-mll/GLUE-baselines
cd GLUE-baselines
python download_glue_data.py --data_dir glue_data --tasks all

The repository is deprecated as a development framework but remains the official baseline and download reference. Fine-tune one predictor per task, write the required TSV files, and submit the nine primary task predictions plus Diagnostic Main through the official platform.

lm-evaluation-harness provides the verified tag glue:

pip install lm-eval
lm_eval --model hf \
  --model_args pretrained=bert-base-uncased \
  --tasks glue \
  --num_fewshot 0 \
  --batch_size auto

Check what that tag expands to in the installed version:

lm-eval ls tasks | grep -E 'glue|cola|mnli|mrpc|qnli|qqp|rte|sst|wnli'
lm-eval validate --tasks glue

Current lm-eval documentation lists CoLA, MNLI, MRPC, QNLI, QQP, RTE, SST-2, and WNLI under the tag, uses validation splits, and does not expose STS-B there. Therefore its output is not an official GLUE Score. Report it as a validation subset with the task list and harness commit.

History & lineage

• 2018-04-20 - GLUE is released as a unified benchmark, diagnostic resource, and online evaluation platform built from nine existing datasets.
• 2018-10-11 - BERT reports 80.5 on the official leaderboard, making GLUE the standard demonstration of transferable pretrained encoders.
• 2019-05 - A conservative human study estimates 87.1. In the same month, the GLUE team releases SuperGLUE because benchmark performance has already passed the non-expert aggregate.
• 2019-07-25 - RoBERTa reports an 88.5 official leaderboard average. This page uses that dated, protocol-specific crossing as GLUE’s saturation point.
• 2020 onward - Research increasingly reports eight-task development averages, omits WNLI, or moves to SuperGLUE and focused diagnostics. GLUE remains useful for regression testing and teaching fine-tuning, but not as a frontier headline.

The lineage is unusually direct: GLUE established the multi-task NLU scoreboard, then its own authors replaced it with SuperGLUE when transfer learning consumed the available headroom.

Further reading

• Wang et al. - original GLUE paper: https://aclanthology.org/W18-5446/
• Official benchmark, FAQ and leaderboard: https://gluebenchmark.com/
• Official baseline and data downloader: https://github.com/nyu-mll/GLUE-baselines
• Nangia and Bowman - conservative human estimate: https://arxiv.org/abs/1905.10425
• Wang et al. - SuperGLUE successor: https://arxiv.org/abs/1905.00537
• Gururangan et al. - NLI annotation artifacts: https://arxiv.org/abs/1803.02324
• McCoy et al. - HANS and syntactic heuristics: https://arxiv.org/abs/1902.01007
• EleutherAI - current validation-only GLUE tasks: https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/glue