benchwiki
safety-alignment

XSTest

Active
2023-08-02 → today

A 450-prompt test of whether models refuse unsafe requests without over-refusing superficially similar safe ones.

last updated 2026-09-14

Performance over time

vendor-reported independent
304050607080901002023

Baseline: No single human score is appropriate. Humans created and labeled the prompts and response taxonomy; the desired direction is compliance on safe prompts and refusal on unsafe prompts.

Overview

XSTest measures a safety failure that attack-only benchmarks miss: exaggerated safety, where a model refuses a benign request because it contains sensitive words or resembles an unsafe request. It pairs 250 safe prompts with 200 unsafe contrasts so evaluators can see both harmful compliance and unnecessary refusal.

This two-sided view makes XSTest useful for calibration. A model that refuses everything is safe only in a trivial sense; it is not a useful assistant.

Task details

The 250 safe prompts cover ten types, including homonyms, figurative language, safe target groups, safe contexts, and benign requests that mention potentially dangerous objects or actions. The 200 contrast prompts use related surface forms but request unsafe assistance.

The model produces a normal free-text response. Evaluators label the response with the paper’s taxonomy, including full compliance, full refusal, partial refusal, and disclaimers.

Example

A safe prompt might ask how to “kill” a running computer process. A keyword-triggered system may refuse because it sees a violent verb; a well-calibrated assistant explains the operating-system command. Its unsafe contrast would use genuinely harmful intent and should be refused.

Scoring methodology

Timeline dating convention: the chart places each result at the underlying model’s public launch date, not the XSTest evaluation date. This shows how models released over time balanced unsafe refusal against safe-prompt over-refusal. Dates were checked against the official GPT-4, Llama 2, and Mistral 7B announcements. The plotted scalar is total refusal on unsafe prompts; each label also gives total refusal on safe prompts.

The official workflow first collects completions, then classifies responses manually or with the repository’s automated scripts. Report safe-prompt refusal and unsafe-prompt refusal separately. Collapsing them into one number requires an explicit weighting of helpfulness against harmlessness and can hide a degenerate refuse-everything policy.

Response taxonomies matter. A disclaimer followed by useful content is not equivalent to a refusal, and partial answers can be policy-dependent.

What it doesn’t measure

XSTest does not cover long conversations, jailbreak optimization, multilingual prompts, multimodal inputs, tool actions, domain-specific policies, or the severity of harmful content. Its unsafe contrasts reflect a general-assistant assumption and may not fit every deployment.

Known issues & criticisms

  • The set is small and intentionally diagnostic rather than representative of production traffic.
  • Public prompts allow benchmark-aware training and prompt-specific tuning.
  • Refusal labels can be ambiguous when a model gives warnings, redirects, or partial information.
  • The desired response depends on application policy; a medical, educational, or enterprise system may reasonably draw different boundaries.
  • Surface-similar pairs expose lexical over-triggering but do not cover every cause of over-refusal.

Contamination notes

All 450 prompts and many paper-era model completions are public. Leakage can improve benchmark-specific behavior, although the paired design still reveals some crude keyword filters. Use XSTest as a regression suite and add private prompts derived from the target product’s real false-refusal reports.

How to run it

git clone https://github.com/paul-rottger/xstest.git
cd xstest

Generate one response for every row in xstest_prompts.csv, then label responses manually or with evaluation/classify_completions.py. The repository’s analysis.ipynb reproduces paper analyses. Record the exact chat template and classifier, because refusal wording is template-sensitive.

History & lineage

XSTest was submitted on 2023-08-02, revised through v3, and published at NAACL 2024. Later work has reused its prompts for response-safety and refusal-classifier evaluation, often under names such as XSTest-Resp. Those derived classifier tasks are related but not identical to evaluating a generative assistant end to end.

Further reading