benchwiki
instruction-following

IFEval

Deprecated

~500 prompts, each carrying an explicit, machine-checkable constraint (format, content, or style) that a response must respect.

last updated 2026-08-12

Performance over time

No performance time series — this benchmark has no cleanly-sourced, metric-consistent aggregate scores to plot. See the prose below for why, and the facts sidebar for what it measures.

Overview

IFEval, introduced by a Google Research team (Zhou et al., November 2023), reframed instruction following as a verifiable measurement problem rather than a matter of human taste. Its construction is deliberately inverted relative to most evals: instead of asking judges “was that answer good?”, it builds prompts whose success criteria are formal enough to be graded by a deterministic, programmatic checker — no human, no LLM-as-judge, no labeler. Each prompt embeds one or more verifiable instructions (such as “write at least three bullet points,” “do not use any commas,” or “respond only in JSON”), and a response scores by whether those precise constraints are satisfied.

Its motivation is squarely about the cost of evaluation. Human-preference evals do not scale: every new prompt needs a fresh paid labeler, and every frontier result is compared through a foggy lens of subjective judgment. IFEval deliberately trades away everything a human reviewer might value — usefulness, tone, nuance, and correctness — in exchange for a score that nobody must pay a labeler to produce. That trade-off is the whole point, and it is narrow by design.

IFEval is now the de facto verifiable-instruction baseline. It is integrated into harnesses such as lm-evaluation-harness as task ifeval, it spawned multilingual and multi-turn descendants (notably Multi-IF from Meta), and it remains a cheap and reliable regression check. In this site’s status system it is now deprecated for frontier comparison: newer suites cover multi-constraint, multi-turn, and agentic instruction-following failures more directly, while IFEval increasingly operates as a pass-fail floor.

Task details

Each item is a prompt that specifies a content task (e.g. “write an email proposing a project,” “summarize a passage,” “produce a list of ideas”) combined with one or more verifiable instructions. The model must produce the requested content and satisfy every stated constraint. The released set is 541 prompts built around 25 types of verifiable instructions, organized into categories such as format constraints (response length, bullet lists, JSON, headers, keyword requirements), language registers (formal vs informal, Shakespearean), and content restrictions (start with a keyword, avoid a keyword, include a specific phrase).

The task is scored at two levels. Prompt-level accuracy asks whether the whole response satisfies all constraints in the prompt. Instruction-level accuracy grades each instruction independently — a response meeting two of three constraints earns partial credit at the instruction level. Each is also reported in a strict and a loose variant (see Scoring methodology). A typical “chat evaluation” config runs zero-shot with temperature 0 on an instruct-tuned model, since base models were never the intended population.

Example

A well-known public instance (key 1001) is the Shakespearean-style, comma-free prompt:

Prompt: “I am planning a trip to Japan, and I would like thee to write an itinerary for my journey in a Shakespearean style. You are not allowed to use any commas in your response.”

The instruction is tagged with instruction_id: punctuation:no_comma. A compliant response writes the itinerary in archaic, Shakespeare-adjacent prose with no comma anywhere; the deterministic checker scans the output and fails the prompt if a single comma appears, regardless of how eloquent the prose. A brilliant response containing one comma fails at the prompt level (though it may still pass other instruction-level checks).

The example shows both the promise and the bias of the design: grammar, eloquence, and usefulness are invisible to the grader; only the absence of commas is measured.

Scoring methodology

The headline numbers are strict and loose accuracy, each computed at the prompt and instruction levels. The checker is a deterministic rule-based component: for each of the 25 instruction types there is a hand-written assertion on the model’s generated text (counting words, regexing for a keyword, detecting a comma, validating an email). “Strict” requires exact compliance; “loose” relaxes the criterion — for example, accepting a keyword appearing anywhere or allowing a missed constraint to not sink the whole response.

Because the grader is exact and stateless, IFEval is essentially free and reproducible to run: no judge to calibrate, no run-to-run variance, no whisper of inter-annotator disagreement. The flip side is that the surface measured is intentionally thin — “following instructions” here means satisfying syntactic constraints, not understanding intent. Reporting should always state strict vs. loose and prompt vs. instruction level together; a bare “IFEval 82” is not comparable. The official repo provides the reference grader for reproduction.

What it doesn’t measure

IFEval measures only whether a model respects a set of concrete, mechanically checkable constraints. It does not measure whether a model honors the spirit of an ambiguous instruction, whether its deliverable is useful or correct, whether it refuses harmful requests, or whether it maintains persona and coherence across a conversation. A model can score 100% on IFEval while being useless — writing an ungrammatical, off-topic, or hallucinated email that merely happens to contain exactly the required bullet points and no commas. Conversely, a genuinely helpful assistant that slightly stretches a formatting rule can score near zero on a prompt. Compliance with declared rules and compliance with unstated intent are different measurements; IFEval only measures the former.

Known issues & criticisms

  • Verifiable ≠ representative. The more objective a predicate is, the less it resembles the real, largely judgment-driven constraints users impose. Critics note that the set favors artificial, checkable constraints over the ambiguous, implicit expectations of actual user work, so a high score does not guarantee real-world follow-through.
  • Preference ≠ compliance. A model can win a human-preference vote while breaking an explicit constraint, because a fluent, on-topic answer “feels” right even when it ignores a formatting rule; conversely a compliant but awkward answer can lose the vote. IFEval scores can therefore contradict preference-based scores for the same model, and neither alone is the truth. The benchmark is explicitly not a proxy for human judgments of quality.
  • Ceiling effect. The instructions are trivial for humans and increasingly passed by frontier models; the set now operates more as a pass-fail floor than a fine-grained army. This is expected and by design, but it limits usefulness for comparing closely-matched top models.
  • English-only and single-turn at the core. The original is English-only and one-turn; the multilingual and dialog turn-length extensions (Multi-IF, M-IFEval) exist precisely to fill what the original knowingly leaves out.

Contamination notes

Risk: low. The instruction set and graders are public, and the paper describes an explicit prompt-level decontamination layer filtering candidates that appear in training data. The set is static — there is no refreshed or held-out version — so prompts can surface in training corpora. But because grading is about satisfying a local, syntactic rule, memorizing the answer text matters far less than in knowledge evals: a model that saw “don’t type commas” has to actually not type commas. Leakage would mainly compress an already-saturated score rather than flip a specific result, which is why the site rates the risk low.

How to run it

lm-evaluation-harness (task ifeval — the most standard route):

pip install lm-eval
# chat / instruct models are the intended population for IFEval
lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-3.1-8B-Instruct \
  --tasks ifeval \
  --log_samples --output_path results/ifeval

This runs the standard strict/loose, prompt/instruction scores and prints them in a per-task table. If you apply the harness’s chat template, keep the template constant across runs — strict scores are markup-sensitive.

Direct from the source repo and raw data:

git clone https://github.com/google-research/google-research.git
cd google-research/instruction_following_eval
# raw input prompts live in data/input_data.jsonl, e.g.
# https://raw.githubusercontent.com/google-research/google-research/refs/heads/master/instruction_following_eval/data/input_data.jsonl

The repository holds the grader and the prompt source. When reporting, always state strict vs. loose, prompt vs. instruction, temperature (use 0), and which variant of the instruction set you ran against.

History & lineage

  • 2023-11 — Zhou et al. post the original IFEval paper on arXiv, proposing verifiable, rule-graded instruction following as an answer to the cost of human preference evaluation.
  • 2024 — Remains an arXiv preprint (not formally published at a peer-reviewed venue); adopted into lm-evaluation-harness as ifeval; becomes the standard cheap baseline scaffold for instruction-following claims across labs.
  • 2024-06 — Meta publishes Multi-IF, extending the IFEval style to unseen input formats, multilingual settings, and controllable feedback loops; M-IFEval similarly broadens the original across languages and turns.
  • Ongoing — IFEval remains useful as an inexpensive, deterministic regression gate, but this wiki treats it as deprecated for frontier instruction-following comparison because it is now more floor than ordering tool.

If your goal is a reproducible, mechanically-scored signal about constraint obedience, IFEval (or Multi-IF for multilingual/low-turn coverage) is the right tool. If you need to reflect real user intent and quality, pair it with a human-preference eval rather than substituting it.

Further reading