benchwiki
reasoning

HellaSwag

Saturated
2019-05-19 → today

Choose the plausible continuation of an everyday event from one real ending and three adversarially filtered machine generations.

last updated 2026-07-25

Performance over time

vendor-reported independent
405060708090100201920202021202220232024human baseline 95.6%

Baseline: Overall test accuracy reported by the authors; random choice among four endings is 25%.

Overview

HellaSwag is a four-choice sentence-completion benchmark for physically situated commonsense. A model receives a short description of an activity or a sequence of how-to steps and selects what plausibly happens next. The contexts come from ActivityNet video captions and WikiHow articles. The correct ending is the continuation from the source; the three wrong endings were generated by language models and selected to be difficult for discriminators.

The benchmark was introduced after BERT nearly closed the gap on its parent, SWAG. HellaSwag’s contribution was not a new answer format but a stronger construction method. Its authors used Adversarial Filtering (AF) to repeatedly train discriminators on temporary splits, find generated negatives that those discriminators could easily reject, and replace those negatives with harder generations. The result occupied what the paper calls a “Goldilocks zone”: the wrong endings often remain topically fluent enough to fool 2019 models while being obviously implausible to people (original paper).

HellaSwag is now saturated as a frontier benchmark, not evidence that general commonsense is solved. OpenAI reported 95.3% for GPT-4 base in a 10-shot setup on a private holdout, nearly matching the authors’ 95.6% human reference, and the official leaderboard stopped accepting submissions in 2024. Meanwhile, most reproducible runs use the public validation split rather than that private holdout.

Task details

Each item contains a context, an activity label, and four candidate endings. Exactly one ending is the recorded continuation; three are machine-generated distractors. The model chooses one index. No explanation, retrieval, or tool use is required.

The released data has 39,905 training items, 10,042 validation items, and 10,003 unlabeled test items (official dataset card). Each validation and test split mixes:

  • In-domain items whose activity label appeared in training.
  • Zero-shot-category items whose ActivityNet or WikiHow activity label did not appear in training.

That “zero-shot” label describes category overlap, not prompting. A zero-shot lm-evaluation-harness run can still contain both split types. The official test leaderboard averaged accuracy over the entire test set and also displayed in-domain, zero-shot-category, ActivityNet, and WikiHow slices.

AF generated many candidate wrong endings, then iterated over random dummy train/test partitions. A discriminator learned each dummy training partition; easy negatives on the dummy test partition were replaced by candidates the discriminator found harder. Repeating this with an ensemble reduced artifacts available to the discriminator used during construction. It did not prove the remaining choices were artifact-free for later model families.

Example

The official project page gives this WikiHow example:

Context: How to catch dragonflies. Use a long-handled aerial net with a wide opening. Select an aerial net that is 18 inches (46 cm) in diameter or larger. Look for one with a nice long handle.

A. Loop 1 piece of ribbon over the handle. Place the hose or hose on your net and tie the string securely.

B. Reach up into the net with your feet. Move your body and head forward when you lift up your feet.

C. If possible, choose a dark-colored net over a light one. Darker nets are more difficult for dragonflies to see, making the net more difficult to avoid.

D. If it’s not strong enough for you to handle, use a hand held net with one end shorter than the other.

The gold continuation is C. All four endings discuss nets, so word overlap is not enough. The correct choice preserves the procedural goal and ordinary physical logic; the generated distractors contain locally fluent but incoherent instructions.

Scoring methodology

Chance accuracy is 25% because every item has four choices. The official supervised leaderboard used ordinary classification accuracy: count one point when the selected ending matches the gold label, then take the mean over items.

For causal language models, lm-evaluation-harness concatenates the processed context with each ending and evaluates all four as multiple-choice continuations. It reports both acc (highest total log-likelihood) and acc_norm (highest log-likelihood normalized by continuation length). This page treats acc_norm as the primary reproducible metric because it reduces the mechanical preference for shorter endings, but the two values are not interchangeable.

The timeline deliberately names each setup. BERT-Large is the authors’ fine-tuned test submission, whereas GPT-4 is a vendor-reported 10-shot private-holdout result. Those points show the benchmark’s historical trajectory, not a controlled scaling curve. Comparisons should match split, shot count, prompt formatting, tokenizer, and whether acc or acc_norm is reported.

What it doesn’t measure

HellaSwag does not test free-form planning, the ability to explain why an event is plausible, factual breadth, dialogue, long-horizon causal reasoning, or action in an environment. It tests recognition among four supplied endings, and recognition can succeed through relative linguistic fit without a reusable world model. Its domains are also narrow: instructional prose and short descriptions of visually grounded activities in English.

Known issues & criticisms

  • Adversarial filtering is model-relative. AF removed features exploited by the generator/discriminator families available in 2019. The paper itself argues that benchmarks must co-evolve with models; a static filtered set cannot remain adversarial indefinitely (Zellers et al., 2019).
  • The wrong choices have a different origin. The gold ending is human source text while all distractors are machine generations. AF makes this provenance distinction harder to exploit, not nonexistent. Later models can learn generation-era style and coherence signals rather than the intended commonsense relation.
  • Some gold choices are only relatively best. Video captions and WikiHow fragments can be noisy, ungrammatical, or underspecified. A distractor may be plausible in isolation even when it is not the recorded next event, so the task rewards matching the source trajectory rather than accepting every reasonable continuation.
  • Protocol drift is large. Fine-tuned classifiers, few-shot API models, and zero-shot causal LMs with length normalization are different systems. A bare “HellaSwag score” hides enough setup detail to make small gaps meaningless.
  • Human-level benchmark performance is narrow. GPT-4’s 95.3% was computed on a private holdout that OpenAI says was close to a validation score explicitly masked from training (GPT-4 Technical Report). This is stronger evidence than a public-validation result, but it does not establish general physical reasoning.

Contamination notes

Risk is high for the evaluation most people actually run. The official 10,003-item test set was distributed without labels, but the 10,042-item validation set includes labels and is the split configured by lm-evaluation-harness. It has been public since 2019, mirrored widely, and used in model evaluation and fine-tuning collections.

The risk is empirically visible. Meta’s Llama 2 paper found a sizeable “not clean” subset in HellaSwag and reported much higher performance on that subset than on clean examples (Llama 2 paper, contamination table). Exact overlap definitions differ across audits, so this is not a universal contamination percentage. It is evidence that public-split results can be inflated by training exposure.

For credible reporting, name the exact split and dataset revision, prefer a genuinely held-out set when available, disclose overlap checks, and do not compare a public-validation acc_norm number directly with the GPT-4 private-holdout result.

How to run it

lm-evaluation-harness uses the verified task flag hellaswag and evaluates the public validation split:

pip install "lm_eval[hf]"
lm-eval run \
  --model hf \
  --model_args pretrained=meta-llama/Llama-3.1-8B \
  --tasks hellaswag \
  --num_fewshot 0 \
  --batch_size auto

Older harness releases accept the same arguments without the run subcommand. The current task configuration declares test_split: null, so this command does not reproduce an official hidden-test submission. Report both acc and acc_norm, plus the harness version or commit.

To inspect the released splits directly:

from datasets import load_dataset

ds = load_dataset("Rowan/hellaswag")
print(ds["validation"][0])

History & lineage

  • 2018: SWAG establishes four-choice commonsense continuation with adversarially generated endings.
  • 2019-05-19: HellaSwag is released as a harder SWAG descendant, expanding from video captions to WikiHow and using stronger generators, BERT discriminators, longer contexts, and iterative AF.
  • 2019-2020: Fine-tuned systems rise from BERT-Large’s 47.3% to ALUM’s 85.6% on the official test leaderboard.
  • 2023-03-14: OpenAI reports 95.3% for GPT-4 base in a 10-shot private-holdout setup, effectively reaching the 95.6% human reference.
  • 2024-11: The official page closes new leaderboard submissions and notes that most users evaluate on validation.

HellaSwag has no single drop-in successor. Its lasting lineage is methodological: use model-in-the-loop or adversarial collection to refresh tests rather than treating one static set as a permanent definition of commonsense.

Further reading