benchwiki
safety-alignment

RealToxicityPrompts

Deprecated
2020-09-24 → today

One hundred thousand web-text prompts for measuring how often language models continue into toxic text.

last updated 2026-09-14

Performance over time

vendor-reported independent
010201820192020

Baseline: No protocol-comparable human baseline exists; toxicity is assigned by Perspective API rather than by humans completing the prompts.

Overview

RealToxicityPrompts is a test bed for neural toxic degeneration: a model begins from a naturally occurring web-text fragment and generates a continuation that may become toxic even when the fragment itself appears innocuous. The authors released 100,000 English prompts sampled from OpenWebText and scored prompts and continuations with the Perspective API.

The benchmark shaped early detoxification research, but it should no longer carry an entire safety claim. It measures one classifier’s view of English toxicity, not the broader safety of an assistant, and both the data and evaluation recipe have been public for years.

Task details

Each item contains a sentence-level prompt, metadata from its source document, and Perspective API toxicity attributes. An evaluator asks a language model to sample one or more continuations, scores them with Perspective, and aggregates over prompts. The paper analyzes both toxic and non-toxic prompt strata.

Example

A prompt may be an unfinished news or forum sentence that contains no obvious slur. The model completes it several times. If even one continuation crosses the configured toxicity threshold, that contributes to the probability-of-toxicity measure; the most toxic continuation contributes to expected maximum toxicity.

Scoring methodology

Timeline dating convention: the chart places each score at the model’s public launch date, not the evaluation date. This shows how models released over time performed on the same RealToxicityPrompts protocol. Launch dates were checked against the official GPT-1, GPT-2, CTRL, and GPT-3 releases. The plotted score is expected maximum toxicity on non-toxic prompts over 25 generations, where lower is safer.

The two best-known measures are expected maximum toxicity and toxicity probability. Both depend on repeated stochastic sampling: more samples create more chances to observe a toxic continuation. Results therefore require the sample count, decoding temperature, top-p/top-k settings, token limit, Perspective model/version, and toxicity threshold.

Perspective is an automated classifier, not ground truth. Treat its score as a model-mediated label and inspect subgroup errors rather than presenting it as an objective measure of harm.

What it doesn’t measure

The benchmark does not test refusals, jailbreak resistance, factuality, cyber or physical harm, tool use, multilingual toxicity, conversational recovery, or human preferences. It also does not show that a model would generate the same content under a deployed chat template.

Known issues & criticisms

  • Perspective API can over-score reclaimed identity terms and text about marginalized groups, so optimizing the metric can suppress legitimate discussion.
  • OpenWebText reflects web-corpus selection biases and is not a representative sample of all user prompts.
  • Results move with decoding and sample count, making cross-paper numbers easy to miscompare.
  • The static public corpus is likely to overlap modern training data.
  • The official repository still describes parts of data processing as forthcoming, which limits turnkey reproduction.

Contamination notes

All prompts are public and derived from public web text, so training overlap is structurally likely. A model may also have seen published completions or detoxification experiments. Report data hashes and prompt IDs, and pair RTP with fresh adversarial prompts when evaluating current systems.

How to run it

git clone https://github.com/allenai/real-toxicity-prompts.git
cd real-toxicity-prompts
conda env create -f environment.yml
conda activate rtp
python -m scripts.run_prompts_experiments --help

The original workflow requires Perspective API access and, for paper-era GPT-3 experiments, model API access. Do not put credentials in result files. Pin the evaluator version and generation settings.

History & lineage

The paper was submitted on 2020-09-24 and published in Findings of EMNLP 2020. It became a standard detoxification benchmark. Realer Toxicity Prompts (RTP-2.0) later expanded the idea with multilingual and adversarial prompts; it is a successor concept, not a score-compatible refresh.

Further reading