WinoGrande

Overview

WinoGrande is a large-scale successor to the 273-question Winograd Schema Challenge. It asks which of two noun phrases correctly fills a blank in a sentence, where the intended answer should require ordinary world knowledge rather than a local grammatical cue. The authors collected roughly 44,000 candidate problems and used AFLITE (Adversarial Filtering for Iterative Training and Evaluation) to remove examples that a committee of representation-based classifiers could solve too easily. The goal was a harder, larger benchmark with fewer machine-detectable annotation shortcuts than earlier Winograd-style sets.

That construction mattered in 2019: the original paper reported 79.1% for its strongest full-data setting against 94.0% estimated human performance. WinoGrande remains a common language-model evaluation, especially in open-model suites, but “WinoGrande accuracy” now names at least two materially different protocols: supervised fine-tuning evaluated on hidden official test labels, and likelihood ranking on the public validation set used by lm-evaluation-harness.

Task details

Each record contains a sentence with one underscore, option1, option2, and an answer label ("1" or "2"). The model must substitute one option at the blank and choose the reading that makes more sense. Although this is commonly called pronoun or coreference resolution, not every blank is literally a pronoun; the operational task is binary cloze completion grounded in commonsense.

Version 1.1 provides the same 1,267 validation and 1,767 test inputs with several training regimes. The official repository names them XS, S, M, L, and XL; the current dataset card lists 160, 640, 2,558, 10,234, and 40,398 training examples respectively, plus a winogrande_debiased configuration with 9,248 training examples. The hidden-label test submission format expects one prediction column per training size, so a score is incomplete unless it identifies the training regime.

Example

From the public winogrande_debiased data:

Sentence: John moved the couch from the garage to the backyard to create space. The _ is small.

Option 1: garage

Option 2: backyard

Gold answer: Option 1, garage

A likelihood-based evaluator builds two completions and selects whichever the model assigns higher probability. This example is useful because syntax permits both completions; resolving what “create space” refers to supplies the intended answer.

Scoring methodology

The official metric is plain accuracy: one point when the selected option matches the label, zero otherwise, averaged across examples. With two options, chance performance is 50%. There is no partial credit and no judge model.

Protocol details dominate comparison:

Official test versus validation. Test inputs are downloadable but labels are withheld for server evaluation. lm-evaluation-harness instead uses the labeled validation split.
Training size. The original benchmark reports separate XS/S/M/L/XL results. A full-data XL result is not comparable to an XS sample-efficiency result.
Fine-tuning versus likelihood ranking. The original setup trains a classifier. lm-eval performs “partial evaluation,” scoring candidate completions by likelihood, as documented in its task README and configuration.
Prompt context. Zero-shot, one-shot, and few-shot decoder-only evaluations are different protocols. Report the number of demonstrations and whether scores are raw or length-normalized.

What it doesn’t measure

WinoGrande tests whether a model can choose between two supplied readings of a short English sentence. It does not measure open-ended explanation, calibrated uncertainty, long-context reasoning, dialogue, multilingual commonsense, or whether a model can retrieve the relevant fact without answer options. Binary choice also permits 50% performance by guessing and can reward weak comparative preferences that would not support a correct free-form answer.

Known issues & criticisms

Filtering is model-relative. AFLITE removes examples predictable from the representations and classifiers used during construction; it cannot prove that all shortcuts are gone for later architectures. The companion AFLITE paper describes the method as reducing measurable dataset biases, not certifying artifact-free data.
Ambiguity and label noise remain. Dataset Cartography, which experiments with WinoGrande, shows that hard-to-learn regions in NLP datasets often include labeling errors while ambiguous examples behave differently from merely difficult ones. A binary gold label can hide genuinely plausible alternative readings.
The evaluation split is often misnamed. The official test labels are private, while lm-eval explicitly evaluates the validation split. Results reported simply as “WinoGrande” can therefore compare different examples.
Crowdsourced language is narrow. Short English schemas reflect annotator assumptions and cultural priors. Correctly selecting a referent is not a complete test of robust, culturally broad commonsense.

Contamination notes

Contamination risk is high for current foundation models. The benchmark is static; its training and validation examples are fully public, and test inputs have been downloadable since release even though their labels are hidden. The lm-eval task declares should_decontaminate: true and uses the sentence as its overlap query, but that only helps when an evaluator has access to the model’s training corpus. It cannot repair an already trained model or detect paraphrased exposure.

Do not transfer contamination findings about the older 273-item Winograd set directly to WinoGrande. For example, the GPT-3 paper found genuine overlap for Winograd; its WinoGrande results are discussed separately. For a modern model, document corpus-level filtering if available and treat public-validation accuracy as memorization-inclusive otherwise.

How to run it

lm-evaluation-harness uses the verified task flag winogrande, the XL configuration, and the public validation split:

pip install lm-eval
lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-3.1-8B \
  --tasks winogrande \
  --num_fewshot 0 \
  --batch_size auto

Inspect the exact registered configuration before publishing a score:

lm-eval ls tasks | grep winogrande
lm-eval validate --tasks winogrande

For the original supervised protocol:

git clone https://github.com/allenai/winogrande
cd winogrande
bash download_winogrande.sh
pip install -r requirements.txt
python eval.py \
  --preds_file ./YOUR_PREDICTIONS.lst \
  --labels_file ./data/dev-labels.lst

Use the official server for hidden test labels. Report split, dataset configuration, training size, shot count, scoring normalization, and harness version.

History & lineage

2011 - The Winograd Schema Challenge proposes carefully paired pronoun-resolution problems intended to resist selectional-preference shortcuts.
2019-07-24 - WinoGrande appears on arXiv with roughly 44,000 crowdsourced candidates and AFLITE-based bias reduction.
2020 - The work appears at AAAI, and the related AFLITE paper generalizes the filtering approach beyond this dataset.
Version 1.1 - The official release exposes XS/S/M/L/XL training subsets, a shared development set, public test inputs, and hidden test labels. Hugging Face additionally exposes the 9,248-example winogrande_debiased configuration.
Current use - Open-model suites commonly use the lm-eval winogrande task: XL data, no fine-tuning, likelihood ranking, validation accuracy. That descendant protocol should not be presented as the original leaderboard protocol.

WinoGrande’s parent is the Winograd Schema Challenge; its variants change training volume and filtering, not the fundamental two-option cloze task.

shots	zero-shot by default in lm-evaluation-harness; original paper fine-tunes on XS/S/M/L/XL training subsets
reasoning	direct
temperature	—
pass@k	pass@1
tools	no
harnesses	WinoGrande official code `winogrande` lm-evaluation-harness `winogrande`

Performance over time