benchwiki
instruction-following

ComplexBench

Nearing saturation

Bilingual benchmark of how models follow instructions when several constraints must hold at once — where stacking constraints is not linear.

last updated 2026-08-12

Performance over time

No performance time series — this benchmark has no cleanly-sourced, metric-consistent aggregate scores to plot. See the prose below for why, and the facts sidebar for what it measures.

Overview

ComplexBench, from Tsinghua’s thu-coai group (Wen et al., July 2024), targets a specific gap in instruction-following benchmarks. Most of them (with IFEval being the most famous) model individual constraint types — a length rule, a format rule, a keyword requirement — but treat each instruction as a bundle of independent checks. Real human requests rarely work that way. A real instruction stacks several constraints that interact: “write a 400-word bullet list of clearly separated issues, but keep it under three items, in a polite tone, and only for the failing departments.” The parts compete, and one may depend on another.

ComplexBench’s core claim is that adding constraints is not linear. A model that happily satisfies each constraint when it appears alone can still fail the moment several must hold at the same time. Earlier work rewarded the ability to satisfy a single type; ComplexBench deliberately makes the composition itself the object of measurement.

It treats instruction following as a reliability problem rather than a single, vibes-based quality judgement: the ground truth is, “how many of the required constraints, in their required dependencies, did the generated text satisfy?” That frame — counting satisfied requirements rather than asking a chat-quality judge — is what makes the numbers legible and the failures actionable.

The benchmark ships a taxonomy of human instructions: 4 constraint types, 19 constraint dimensions, and 4 composition types describing how constraints combine within a single instruction. To score it, the authors decompose each instruction into a set of per-constraint verification questions, grade each with rules where possible and an LLM judge otherwise, then combine the answers through the dependency structure implied by the composition types. The headline finding is that frontier models still fall well short on the composed-constraint checkpoints even where the same constraints in isolation were manageable.

Task details

Each dataset instance is a single instruction written so that several constraints must all be satisfied at once. The instance carries structured metadata meant to make grading deterministic:

  • instruction / instruction_en — the task in Chinese and, since a January 2025 release, an English translation.
  • task_types — the kind of writing task (e.g. professional writing).
  • constraint_dimensions — the constraint slots in play, chosen from the 19 dimensions (e.g. Length, Helpfulness, Bullets Format, Factuality).
  • composition_types and category — how those constraints combine (a conjunction such as “must do A, B, and C” versus a Selection/branching structure, etc.).
  • scoring_questions — the gold scaffold for grading: each question targets one constraint dimension or composition, has a rule field that is usually null but carries an explicit rule (e.g. model_length:[300,10000]) when the constraint is programmatically checkable, and a dep list declaring which other scoring questions it depends on.

The model’s job is to read the instruction and produce text that satisfies everything, branches and preconditions included. Output format is uncontrolled free-form: the grading questions — not the prompt — define what counts as a pass. The benchmark ships 1,150 English test instructions (the figure this page’s dataset.size matches, per the size-matches-primary convention), with a larger Chinese set, and those instructions expand into roughly 5,306 gold scoring questions that drive the grade. The dependency field (dep) is the structural key to the composition: a scoring question only earns credit if the earlier questions it lists as prerequisites were met. A model that answers “no SDR, no article” for a country is then correctly not credited the 300-character article point for that branch — because the branch requiring an article was never granted.

Example

Illustrative, not a dataset item — a composed-instruction resembling the benchmark’s selection-and-conjunction style. Constraint stack: branch on a condition, then enforce a list format, a tone, a length rule, and a factual ground:

Instruction: For each country listed, first state whether it holds Special Drawing Rights (SDRs). If it does, write an article — at least 300 characters, using bullet points — arguing for its withdrawal application; if it does not, explain why in at most 100 characters.

Constraint dims: Length, Helpfulness, Bullets Format, Factuality — composed as And + Selection.

A strong response branches correctly for each country, then within each branch satisfies the length, format, and factuality constraints that apply to that branch. A model that blurs the two branches — or writes 350 characters of prose without bullets for a “no SDR” case — violates two constraints at once, even though, asked for format only or length only in isolation, it would have passed both. This is the non-linearity ComplexBench is built to expose.

Scoring methodology

The headline number is a percentage of the total verifiable “scoring-credit” satisfied, rolled up through each instruction’s dependency graph.

  • Rule-based checks where possible. Some constraints are trivially machine-verifiable — length is scored by a rule like model_length:[1,100], giving a deterministic pass/fail.
  • LLM judge for the rest. Anything not rule-checkable (helpfulness, factual plausibility on the SDR case, logical soundness) goes to a GPT-4-based judge on a question-by-question basis. Augmenting the LLM judge with explicit rules is the architecture’s hedge against judge drift: rules pin the mechanical constraints while the judge resolves the semantic ones.
  • Dependency-weighted aggregation. Because each scoring question carries a dep list of prerequisites, the aggregate is not a simple mean. A question is only scored when every question it depends on was satisfied, so one early error cascades: if the model misclassifies whether a country holds SDRs, every downstream question tied to that branch (article length, bullets, factual soundness) fails too. This is how the paper turns “did the model respect the composition” into a number rather than a set of independent coin-flips.

Grading thus depends on the judge quality on those semantic questions and on the correctness of the dependency labels. Scores across papers are only comparable when the same evaluation protocol (judge model, rules, dependency resolution) is used.

What it doesn’t measure

ComplexBench measures verifiable constraint coverage under composition — the reliability of instruction following. It does not measure the natural-distribution quality of the response at large such as a reward-model preference, nor the style of writing beyond what a constraint names. Because it is a self-contained compositional task, it does not measure open-ended long-horizon behavior, tool use, retrieval, or agentic multi-step persistence — there is no environment to interact with, only a single prompt to satisfy in one response. And despite being bilingual (English + Chinese), it does not claim to sample the full diversity of human instruction distributions or non-writing domains.

Known issues & critiques

  • LLM-as-judge for the semantic checks. Rules remove mechanical noise on the mechanical constraints, but the unresolvable dimensions (helpfulness, factual plausibility) rest on GPT-4-as-judge. A judge can be systematically lenient, and judging may not be even across the two languages. The authors mitigate this with rule augmentation, but the overall score still reflects one judge rather than a hand-graded ground truth.
  • Dependency errors amplify. Because one failed prerequisite zeroes out all downstream questions, a single judge error at the base of a dependency chain becomes a multiplied penalty. Aggregate scores are therefore sensitive to how well the dep labels and the judge’s early decisions are calibrated, not just to the model’s output quality.
  • Manual curation, not an attack stress-test. The dataset is hand-written synthetic and controlled — good for contamination control, but it is not an ecological sample of real user instructions, so its difficulty may not transfer to production traffic.
  • Public gold questions. The full grading schema (scoring questions, rules, dependencies) ships inside the dataset file itself. That is what makes evaluation reproducible, but a fine-tune that has seen the scoring questions learns exactly what the grader checks — a structural leak vector that is not hypothetical.

Contamination notes

Risk: low relative to web-crawled benchmarks. Items are mostly original synthetic compositions authored for the benchmark rather than mined from the internet, so the classic training crossed the answers in a crawl path is limited. Because items are compositional, memorizing a single right answer is a weak attack — a model still has to actually obey the composed constraints to score. The gold scoring questions are public in the repo, however, with no held-out set, so fine-tuning on the released files is a real vector even at the top end. That static, public, non-refreshed data is the structural ceiling on how long the benchmark can stay trustworthy.

How to run it

Use the official repository’s two-step flow — generate responses yourself, then delegate scoring to the repo’s eval.sh script. Setup:

git clone https://github.com/thu-coai/ComplexBench.git
cd ComplexBench
pip install -r requirements.txt

Step 1 — generate responses. The repo does not host a generator; you query your target model on the instructions in data/data_final.json and write one JSONL per model (see llm_generations/* for the format, replacing the response field with your own). The authors recommend greedy decoding (temperature 0) to reduce variance. The repo ships the responses it generated for the 15 LLMs in the paper.

Step 2 — evaluate. Run the provided script; it takes an OpenAI API key for the judge pass (the README signature passes the same key twice, alongside a model name and the language):

bash eval.sh "data/data_final.json" "llm_generations/glm4.jsonl" \
  "evaluation_results" "<OPENAI_API_KEY>" "<OPENAI_API_KEY>" \
  "glm4" "zh"

Results: evaluation_results/glm4_final_results.json holds each scored question’s answers; evaluation_results/glm4_statistics.json holds the aggregate statistics. Pass en as the language to score the English-translated data added in the 2025-01-29 release. This is the only official harness command the repo documents — there is no automated leaderboard export.

History & lineage

  • 2024-07-04 — Paper posted (arXiv:2407.03978); dataset and evaluation scripts released on GitHub under MIT.
  • 2024-07-11 / 2024-10-31 — Paper revisions on arXiv.
  • NeurIPS 2024 — Accepted to the Datasets & Benchmarks Track, marking it as a methodological contribution — how to score composition — rather than another large MCQ set.
  • 2025-01-29 — The project released the English-translated version of the full dataset (data_final.json field instruction_en), making it usable by monolingual-English teams.
  • The framing builds on the decompose-and-LM-judge idea of prior work such as InFoBench, but adds the property that sets it apart: it models the dependencies between constraints rather than treating each as an independent yes/no. No successor has been announced; the benchmark is actively maintained, and its contribution to composition scoring remains novel.

Further reading