benchwiki
coding

HumanEval+

Saturated
2023-05-02 → today

HumanEval's 164 Python tasks re-evaluated with roughly 80 times more tests to expose plausible but incorrect programs.

last updated 2026-08-03

Performance over time

vendor-reported independent
70809020232024

Baseline: No controlled aggregate human pass@1 baseline has been published for HumanEval+; human-written canonical solutions are test oracles, not a timed baseline.

Overview

HumanEval+ keeps HumanEval’s 164 prompts but replaces “passes a handful of tests” with a much stronger approximation of functional correctness. EvalPlus automatically produced inputs using LLM-assisted and mutation-based generation, validated them against canonical implementations, and expanded test coverage by roughly 80×.

That change matters: in the original study, stronger tests reduced reported pass@k by as much as 19.3–28.9 points and could reverse model rankings. HumanEval+ is therefore the preferred member of this lineage, although its small public prompt set is now saturated as a frontier-ranking benchmark.

Task details

Generation is identical to HumanEval: complete a short Python function from its signature and docstring. Evaluation runs each completion first against the original base tests and then against the enlarged “plus” tests. The paired outputs reveal whether code that appears correct under HumanEval survives edge cases and broader input coverage.

Example

For HumanEval/58, the EvalPlus paper shows a ChatGPT-generated implementation that passed the original tests but failed additional generated inputs; on that task, the measured pass rate fell by 28.5% under the new tests. The failure pattern is representative: code captures the examples’ apparent rule while missing inputs permitted by the specification.

Scoring methodology

The headline score is task-level pass@1 under all HumanEval+ tests. EvalPlus also reports base pass@1 for the same generations under original tests. That pairing controls for generation differences and isolates evaluator strength. A completion receives no partial credit: syntax errors, exceptions, timeout, or any failed assertion fail the task. Dataset version, timeout, Python version, prompt mode, sanitizer, and backend must be recorded.

What it doesn’t measure

More tests do not broaden HumanEval’s task distribution. HumanEval+ still measures short Python functions, not large codebases, multi-file work, library discovery, maintainability, security, or developer interaction. It evaluates outputs, not the validity of their reasoning, and does not measure computational efficiency except through timeout behavior.

Known issues & criticisms

Generated tests are only as correct as the task contract and canonical oracle. EvalPlus has issued multiple contract, input, and oracle fixes, so scores from different dataset versions can differ. Test execution also runs untrusted code and requires isolation. Finally, stronger tests reduce false positives but cannot eliminate specification ambiguity or contamination of the underlying prompts.

Contamination notes

Risk is high. HumanEval+ adds difficult public tests; it does not create new private problems. Models may have encountered HumanEval prompts, canonical solutions, EvalPlus tests, or code trained specifically to pass them. Use the benchmark as a regression test and pair it with fresh or private coding tasks for generalization claims.

How to run it

EvalPlus can generate and evaluate in one command, with Docker recommended for execution:

python -m pip install --upgrade evalplus
evalplus.codegen --model gpt-4o-2024-08-06 \
  --dataset humaneval --backend openai --greedy
evalplus.evaluate --dataset humaneval --samples path/to/generated_samples.jsonl

For local samples, mount only the results directory into the project’s Docker image as documented by EvalPlus. Report both base and plus pass@1, the EvalPlus version, prompt mode, model snapshot, and backend.

History & lineage

EvalPlus and HumanEval+ appeared on 2023-05-02 and were published at NeurIPS 2023. Subsequent releases corrected task contracts, inputs, and oracles and introduced a reduced HumanEval+ Mini. The project later added MBPP+ and EvalPerf, extending the same idea from correctness coverage to another dataset and then to code efficiency.

Further reading