benchwiki
coding

HumanEval

Saturated
2021-07-07 → today

OpenAI's 164 hand-written Python function-synthesis tasks, graded by executing generated completions against unit tests.

last updated 2026-08-03

Performance over time

vendor-reported independent
20304050607080901002021202220232024

Baseline: No controlled aggregate human baseline was published; the tasks were hand-written and reference solutions were authored by people, but that is not a timed pass@1 measurement.

Overview

HumanEval is the benchmark that made executable functional correctness the standard language of code-model evaluation. OpenAI released 164 hand-written Python problems with Codex: each supplies a function signature and docstring, and the model completes the body. A completion succeeds only if it runs and passes the supplied tests.

HumanEval is now saturated as a frontier ranking tool. This does not mean every generated program is reliable. The set is small and public, and the HumanEval+ study demonstrated that its sparse tests accepted many wrong solutions. HumanEval remains useful for reproducing historical results and smoke-testing a pipeline; use HumanEval+ for stricter correctness.

Task details

Each JSONL record contains a task_id, prompt, canonical solution, tests, and entry point. The prompt ends inside a Python function whose docstring describes behavior and often includes examples. The model returns only the completion. The harness appends tests, executes the program with a timeout, and records pass, failure, or timeout.

Example

HumanEval/0 asks for a function that decides whether any pair of numbers in a list is closer than a threshold. A correct completion checks all distinct pairs and returns True on the first qualifying distance. A tempting wrong answer checks only adjacent list elements; it can pass friendly examples but fails when the close pair is separated in the input order.

Scoring methodology

For each task, sample n completions and count c that pass every test. The paper uses an unbiased estimator for the probability that at least one of k draws succeeds, reported as pass@k. Pass@1 should not be confused with pass@100: repeated sampling supplies much more search. Temperature, sample count, prompt wrapper, stop sequences, Python version, timeout, and completion cleanup all affect the result.

What it doesn’t measure

HumanEval covers short, self-contained Python functions. It does not measure repository navigation, API use, patch integration, security, maintainability, code review, interactive debugging, multi-file changes, or efficiency beyond avoiding a timeout. Passing tests establishes agreement on tested inputs, not proof of correctness.

Known issues & criticisms

EvalPlus generated far more tests and found that original HumanEval could overstate pass@k by up to 19.3–28.9 percentage points across evaluated models, sometimes changing rankings. The public issue tracker also documents disputed tests and canonical solutions. Finally, 164 tasks provide a noisy ceiling-region comparison, and many reports called “HumanEval pass@1” use incompatible chat prompts or greedy decoding rather than the paper’s sampling protocol.

Contamination notes

Risk is high. HumanEval was hand-written to be clean relative to Codex’s training corpus in 2021, but the complete benchmark—including tests and reference implementations—is public and widely copied into repositories, papers, tutorials, and model-development workflows. That original construction safeguard does not protect later models.

How to run it

Run generated code only inside a strong sandbox; the official repository explicitly warns that model completions are untrusted:

git clone https://github.com/openai/human-eval.git
python -m pip install -e human-eval
evaluate_functional_correctness samples.jsonl

Each line of samples.jsonl must contain task_id and completion text. Generate enough samples for the reported k, and record decoding settings and Python environment.

History & lineage

OpenAI introduced HumanEval with Codex on 2021-07-07. The paper’s Codex-12B solved 28.8% at pass@1 and 70.2% with 100 samples. HumanEval became the default compact code-generation test, then inspired stricter tests in HumanEval+, translations in MultiPL-E, and many prompt and mutation variants. EvalPlus’s 2023 audit is the clearest point at which the original tests ceased to be adequate for frontier claims.

Further reading