benchwiki
instruction-following

MT-Bench

Nearing saturation
2023-06-09 → today

80 curated multi-turn questions across 8 skills, each a two-turn conversation scored by a strong LLM judge on a 1–10 scale.

last updated 2026-08-07

Performance over time

vendor-reported independent
0102023

Baseline: No canonical human score on the 1–10 judge scale has been published. The paper instead validated the judge against human preference annotations: GPT-4-judge agreement with human preference exceeded 80%, roughly matching human–human agreement on the same annotations. That is an agreement check, not a human competence score.

Overview

MT-Bench (“Multi-turn Benchmark”) asks a question that push-button QA evals could not: does a model actually follow and stay helpful inside a conversation? Most benchmarks present one turn and one task. MT-Bench poses 80 real, hand-curated questions across 8 categories — writing, roleplay, reasoning, math, coding, extraction, STEM, and humanities/social science — and asks the model to answer each one twice: respond, then handle a follow-up that explores ambiguity, correction, or another request from the previous turn. A strong LLM (originally GPT-4) then grades both answers on a 1–10 scale.

Its importance is twofold. First, it was the concrete demonstration that an LLM can stand in for human evaluators: the paper found the GPT-4 judge agreed with human preference over 80% of the time, about the same as human annotators agreed with each other. That single result is the evidence behind the “LLM-as-a-judge” practice that now underpins scores across much of the field. Second, MT-Bench sits as the scripted companion to Chatbot Arena (arXiv:2403.04132) — the same LMSYS group’s crowdsourced pairwise platform — covering the two common ways of measuring helpfulness: a fixed, labelable test set (MT-Bench, an LLM judge scored a 1–10 scale), and live human preference ranking (Chatbot Arena, where users vote between two anonymous model responses and an Elo-style ranking emerges from those votes). They deliberately cover different problems: MT-Bench is reproducible and cheap but frozen; Arena is always-current and measures what real users prefer, but it is a popularity/El rating and cannot label why one response beats another.

Task details

Each of the 80 questions is a scripted two-turn dialogue. In turn 1, the model receives a category-labelled prompt it must answer usefully. In turn 2, it is given the same thread with a follow-up — a clarification, a request to redo under different constraints, a new but related ask. The model is scored on the whole thread: does it answer both turns well, and does turn-2 behaviour build coherently on turn-1 output? Categories are not equally represented — reasoning, math, and coding questions are scarcer (often 4 each on the canonical set) than the conversational ones.

The input is the raw dialogue history; the output is free-form text per turn. There is no option set, no sandbox, and the evaluator cannot inspect execution: a “math” or “coding” answer is judged on the prose the model produces, not on whether printed code would run. This is a deliberate design choice for a helpfulness/quality eval, but it is also the main thing the benchmark does not measure (see What it doesn’t measure).

Example

An illustrative in-style two-turn item (constructed to match MT-Bench’s format, not drawn from the 80):

Turn 1 — roleplay: “You are a sarcastic but competent science tutor. Explain why the sky is blue, but don’t give the ‘scattering’ answer in your first paragraph.”

Good model turn-1: “Oh, the Sky glorious Sky — the reason it never submits to your fashionable preference for a navy wallpaper. At its most literal, it’s a backdrop question: what does the whole sky look like, and why do human eyes read it as blue?” (then the actual Rayleigh-scattering explanation in later paragraphs.)

Turn 2 (follow-up): “Wait, but why does it look orange at sunset, not blue? Answer that without ignoring what I said about scattering.”

Good model turn-2: “Because sunlight has to travel through more atmosphere at the horizon. The short blue wavelengths get scattered away sidewise along that longer path, leaving the longer red/orange ones; the scattered away contributions to the deep blue.”

A weak turn-2 instead repeats the Rayleigh-scattering point from turn-1, or answers the sunset question without referencing the constraint it set. The judge rewards a follow-up that reuses the thread, respects the added instruction, and stays helpful — the multi-turn coherence that single-turn scaffolds miss.

Scoring methodology

The primary number is the mean judge score on 1–10, averaged across the two turns of all 80 items. The scoring engine is an LLM — originally GPT-4 behind an API call that feeds the judge a rubric prompt and the entire dialogue. (See also the win-rate variant in the spec sheet, the other common reporting convention.) Because the judge is a model rather than a rule, the score inherits judge biases (below), and because the judge is a chosen model, scores are only meaningfully comparable between runs that use the same judge.

Two design decisions matter: single-turn judging (each turn scored separately, with a “sanity check” prompt step to catch judge misbehaviour) and the absence of cross-example interaction — there is no whole-thread global ranking, unlike Chatbot Arena’s ordinal wins. MT-Bench also surfaces the judge’s own prose as a rationale (“judge reasoners”), which lets the human reviewer inspect why a score was assigned rather than trusting an opaque number.

What it doesn’t measure

MT-Bench measures a judge-rated impression of multi-turn helpfulness, not the external quality of the output: code is assessed on the page rather than executed, math answers are not checked for correctness by a solver, and there is no tool execution, sandbox execution, or grounding. With category sizes of ~4–8 questions, it offers no reliable per-skill diagnostic (its own paper cautions against category-level conclusions), and with 80 items it is not a statistical instrument for fine-grained capability. It does not measure safety or refusal behavior — a safe-but-unhelpful model and a genuinely harmful-but-confident mode are both judged on helpfulness. It is a judge-preference surrogate, not a collection modern label verifiable.

Known issues & criticisms

  • Judge bias that the paper itself documented. The paper and subsequent work catalogued position bias (the judge tends to prefer a shown answer), verbosity/length bias, self-preference (a model-judge favours its own style), and sycophancy of the evaluated model toward the judge. Because MT-Bench’s whole number is “did GPT-4 say this was good,” these biases create reproducible but not objective artifacts. The paper mitigates several via easing — e.g., swapped-answer judging — but length bias is known to inflate scores of verbose models.
  • A tiny, fixed question set that is now highly familiar. 80 curated items with heavy public release and near-saturation for frontier models (the best models cluster at the 8.5–9.0 region, where more re-scoring costs more records than it gains). Saturation is not the same as “solved”: the judge limitations, not the model, are the binding constraint at the top.
  • The judge step itself is a moving target. The intended original GPT-4 judge has been mode-length sensitive, and many later numbers report GPT-4-family judges but do not pin the exact model. Cross-paper MT-Bench numbers are not directly comparable unless the judge, its prompt, and the length-limit are all identical.
  • High-end results favour verbose output. The verbosity bias means a model that writes more will tend to score higher simply because it gives the judge more to reward — knowledge that many reported scores maximise unwritten length, which cuts into what a raw MT-Bench number claims about quality.
  • Preference is not constraint compliance. A winning answer is one the judge likes, not one that provably obeys a strict instruction constraint. MT-Bench scores should not be read as consent to instruction-following, which requires a different, more checkable eval.

Contamination notes

Risk is medium (not high) for a specific reason: MT-Bench’s public questions leak into web crawls and training data, and there is no refreshed or held-out version. The listed mitigation is real — it reduces the memorisation-mode of the eval, since a model that has memorised an answer still has to produce it in the right two-turn shape and survive re-annotation by a same judge — and the 80 items have been used heavily in instruction-tuning and alignment mixes. But the deeper issue is the size: with such a small set, and the same 80 items everywhere, the benchmark is effectively a fixed public calibration target rather than a moving goal. As with any static public eval, treat a “score of 8.99” as a certificate of “tested on this exact set,” not as a certificate of general helpfulness.

How to run it

Fast & copy-pasteable, using the FastChat llm_judge scripts, which generate answers and then judge them with a selected model:

git clone https://github.com/lm-sys/FastChat.git
cd FastChat/fastchat/llm_judge
python -m pip install fschat   # or `pip install -e [llm_judge]` from repo root

# 1) generate the two-turn answers for your model
python gen_model_answer.py --model-path meta-llama/Llama-3.1-8B --model-id mymodel

# 2) have a judge (default gpt-4) score every answer you just generated
python gen_judgment.py -m mymodel

# 3) display the mean 1–10 scores (and see per-category + the reasoning)
python show_result.py -m mymodel

Then follow the printed show_result table. The judge prompt and rubric live in judge.py, and gen_judment.py orchestrates the judge calls. If you use the lm-eval harness instead, it exposes an mt_bench task for generation and then requires a judge step. Always report: the judge model, the exact judge prompt version, and the prompt to generate — a bare “MT-Bench 8.4” is not interpretable without all three.

History & lineage

MT-Bench was introduced in the June 2023 paper (arXiv:2306.05685), accepted at NeurIPS 2023, and authored by researchers at UC Berkeley, UC San Diego, Stanford, CMU, and MBZUAI. It shipped alongside Chatbot Arena (arXiv:2403.04132), with which it argues the same thesis from different angles: MT-Bench is the organised, judged set; Arena is the live, human pairwise ranking platform. Together (with the later Arena-derived ranking) they became the default “quality-vibe” gauges for chat models.

What changed since: the Winogard “win-rate” and mean-score reporting diverged, many labs now only publish a single “MT-Bench” judge score without pinning the judge; and MT-Bench remains the entry point to the LLM-as-a-judge technique it helped validate. There is no direct successor; more recent LLM-judge-and-better-call-suites (e.g., LongBench for length, or the various “judge-LLM” training suites) are footnotes, not single drop-in replacements. MT-Bench is best treated today as a standard, fixed calibration — not a moving evaluation target.

Further reading