benchwiki
safety-alignment

WildGuardTest

Active
2024-06-26 → today

A human-annotated test set for prompt harmfulness, response harmfulness, and refusal classification.

last updated 2026-09-14

Performance over time

vendor-reported independent
6575859520232024

Baseline: The labels come from three independent annotators per prompt-response pair, but no single human model-style F1 baseline is reported.

Overview

WildGuardTest is the held-out portion of WildGuardMix for evaluating safety moderators. Unlike a harmful-prompt-only set, it supports three linked decisions: whether a user prompt is harmful, whether an assistant response is harmful, and whether the response is a refusal. It mixes straightforward prompts with adversarial jailbreaks and includes benign as well as harmful interactions.

The benchmark evaluates classifiers, not only chat models. WildGuard is the companion 7B moderator trained on WildGuardTrain; WildGuardTest is the test set and should not be used for training.

Task details

Each of the 1,725 rows contains a prompt, an optional response, an adversarial flag, and available majority labels for prompt harmfulness, response harmfulness, and response refusal. The examples span 13 risk categories and draw from synthetic data and in-the-wild interactions.

Three independent annotators label each prompt-response pair. Items without agreement for one field can retain labels for another, so task denominators may differ.

Example

A sanitized pair can contain a harmful request followed by a clear refusal. A correct moderator labels the prompt harmful, the response unharmful, and the response as a refusal. Treating the entire conversation as one harmful/not-harmful bit would lose that distinction.

Scoring methodology

Timeline dating convention: the chart places each score at the moderation model’s public launch date, not the WildGuardTest evaluation date. This lets the timeline show how moderation models released over time performed on WildGuardTest. Dates were checked against the official GPT-4 announcement, Llama Guard 2’s Llama 3 release record, and the WildGuard paper release. Only total prompt-harmfulness F1 is plotted; response-harmfulness and refusal F1 remain separate metrics in the paper.

The paper reports F1 for each task and examines vanilla versus adversarial subsets. Accuracy alone can obscure class imbalance, while one average across the three tasks hides whether a classifier recognizes harmful intent, harmful output, or refusal wording.

WildGuardTest’s gold labels are human majority decisions audited with a prompted GPT-4 classifier. Inter-annotator agreement is moderate to substantial rather than perfect, so small model gaps should not be over-interpreted.

What it doesn’t measure

WildGuardTest does not measure whether a model can safely complete a complex user task, whether a refusal is helpful, downstream tool effects, multilingual moderation, policy customization, or emerging harms outside its 13 categories.

Known issues & criticisms

  • Safety labels depend on the dataset’s policy taxonomy and may not match a product’s rules.
  • The English-only set underrepresents cultural and linguistic variation.
  • Refusal detection is not equivalent to safety: a refusal can leak harmful details, and a compliant answer can be benign.
  • Public benchmark use can lead to classifier overfitting.
  • Moderate annotator agreement places a practical ceiling on very small claimed gains.

Contamination notes

The test set is released through a gated Hugging Face dataset under Ai2 responsible-use terms, but it is still accessible and static. Gating reduces casual misuse, not training overlap. Keep WildGuardTest out of training mixtures and pair it with fresh, policy-specific evaluation data.

How to run it

git clone https://github.com/allenai/safety-eval.git
cd safety-eval
conda create -n safety-eval python=3.11
conda activate safety-eval
pip install -e .
pip install -r requirements.txt

Then run the three classifier tasks together:

python evaluation/eval.py classifiers \
  --model_name WildGuard \
  --tasks wildguardtest_prompt,wildguardtest_response,wildguardtest_refusal \
  --report_output_path ./classification_results/metrics.json \
  --save_individual_results_path ./classification_results/all.json

Accept the dataset’s gated responsible-use terms before loading it.

History & lineage

The WildGuard paper was submitted on 2024-06-26, revised to a NeurIPS 2024 camera-ready version, and released alongside WildGuardMix, the WildGuard moderator, and the Ai2 Safety Tool. WildGuardTrain and WildGuardTest are distinct splits. XSTest-Resp is a related response-classification set used for cross-benchmark evaluation.

Further reading