benchwiki
safety-alignment

StereoSet

Deprecated
2020-04-20 → today

English context-association tests that separate stereotypical preference from basic language-modeling ability.

last updated 2026-09-14

Performance over time

vendor-reported independent
6575201820192020

Baseline: No protocol-comparable human ICAT baseline is reported. ICAT 100 describes an idealized model with perfect language modeling and stereotype score 50, not measured human performance.

Overview

StereoSet measures whether a pretrained language model prefers stereotypical associations while controlling for whether it can recognize meaningful language at all. It covers four domains: gender, profession, race, and religion. The core idea is important: a model that assigns equally poor probability to every sentence is not fair or useful, so bias and language-modeling ability must be reported together.

StereoSet is best treated as a historical diagnostic. Its examples reflect crowdsourced US social associations, the data are static and public, and the original likelihood-ranking setup was designed for pretrained language models rather than instruction-tuned chat systems.

Task details

The benchmark has two tracks. In the intrasentence track, a model ranks three words or phrases that fill a blank in one sentence. In the intersentence track, it ranks three possible continuations of a short context. Each item offers a stereotypical option, an anti-stereotypical option, and an unrelated option.

The released validation data contain 2,106 intrasentence and 2,123 intersentence contexts. Each context has several candidate sentences, which is why descriptions also refer to roughly 17,000 sentences.

Example

A sanitized intrasentence item can mention a profession and ask the model to rank three completions: one repeats a familiar stereotype, one counters it, and one is nonsensical. A useful unbiased model should rank both meaningful completions above the unrelated one without systematically preferring the stereotype.

Scoring methodology

Timeline dating convention: the chart places each benchmark score at the model’s public launch date, not the date the StereoSet evaluation was run. This lets the timeline show how models available over time performed on StereoSet. Launch dates were checked against the official BERT release, GPT-2 release, and XLNet paper/release; score links point to the StereoSet paper.

Language Modeling Score measures how often the model prefers either related option over the unrelated distractor. Stereotype Score measures how often it prefers the stereotypical option over the anti-stereotypical one; 50 is the neutral target. ICAT combines the two, rewarding language ability while penalizing distance from 50.

Do not report ICAT alone. Two systems can reach a similar ICAT through very different combinations of language competence and stereotype preference, and intrasentence and intersentence scores are separate protocols.

What it doesn’t measure

StereoSet does not measure generated harms, refusal quality, downstream allocation decisions, intersectional bias, multilingual behavior, or whether an assistant follows a deployment policy. It probes relative sentence likelihoods for a fixed set of associations.

Known issues & criticisms

  • The dataset card notes that stereotypes are grounded in US contexts and that its crowdworker population is not demographically representative.
  • Some anti-stereotypical options are implausible or merely invert a stereotype, so neutrality can be conflated with factual or linguistic oddness.
  • ICAT is a composite. It can reward a model for lowering confidence in meaningful text rather than for making a clean fairness improvement.
  • Public static examples and benchmark-specific scoring make modern chat-model comparisons vulnerable to contamination and prompt-format effects.

Contamination notes

The examples, labels, and evaluation code are public. This enables exact reproduction but creates high leakage risk for models trained after 2020. Memorization is especially relevant because every item has a small fixed candidate set. Use StereoSet for regression testing, not as a secret held-out safety claim.

How to run it

git clone https://github.com/moinnadeem/StereoSet.git
cd StereoSet

Follow the repository instructions for the intrasentence or intersentence scorer, or load McGill-NLP/stereoset with Hugging Face Datasets. Hold the tokenizer, likelihood normalization, and track constant across systems. Chat-generation accuracy is not automatically comparable to the original likelihood protocol.

History & lineage

StereoSet was submitted on 2020-04-20 and later published at ACL-IJCNLP 2021. It helped establish the practice of reporting bias together with language ability. Later datasets such as CrowS-Pairs, BBQ, and Contextual StereoSet target different weaknesses; they are related evaluations, not drop-in score-compatible revisions.

Further reading