Overview
MBPP+ applies EvalPlus’s stronger-testing method to MBPP. It keeps basic Python synthesis prompts but evaluates completions with roughly 35× more tests, designed to expose programs that merely fit the original handful of assertions. The current benchmark contains 378 tasks after a later pre-v0.3.0 update removed 21 broken items.
MBPP+ is the recommended MBPP-family evaluator, but it is nearing saturation because the remaining tasks are short, public, and elementary. Its best use is paired reporting: run one set of generations and show both base and plus scores.
Task details
The model receives an MBPP task description plus the harness’s prompt examples and returns Python code. EvalPlus executes the output against original base assertions and a much larger set of generated and validated plus inputs. Tasks preserve MBPP’s focus on small functions, data structures, arithmetic, strings, and basic algorithms.
Example
Consider an MBPP task that asks whether one list is contained in another. A superficial solution may compare sets, incorrectly discarding multiplicity and order constraints while still passing three examples. Additional MBPP+ inputs exercise duplicates, empty cases, and adversarial arrangements, rejecting that implementation when it violates the natural-language contract.
Scoring methodology
Pass@1 is the percentage of tasks whose single completion passes every MBPP+ test. EvalPlus also runs the same output against base tests. This paired design distinguishes generation quality from evaluator strength. Results must specify the dataset version: the transition from 399 to 378 tasks was not a cosmetic change and was expected to increase pass@1 by about four points.
BenchmarkList sampled these provider-sourced scores on 2026-07-13. Its page does not identify the exact EvalPlus dataset revision, so the values remain a labeled snapshot. Per the requested timeline convention, each score is plotted at the model’s official launch date; that date is not an evaluation timestamp.
| Rank | Model | MBPP+ pass@1 | Model launch date |
|---|---|---|---|
| 1 | O1 Preview (Sept 2024) | 80.2% | 2024-09-12 |
| 2 | O1 Mini (Sept 2024) | 78.8% | 2024-09-12 |
| 3 | Qwen2.5-Coder-32B-Instruct | 77.0% | 2024-11-12 |
| 4 | DeepSeek-Coder-V2-Instruct | 75.1% | 2024-06-17 |
| 5 | Gemini 1.5 Pro 002 | 74.6% | 2024-09-24 |
What it doesn’t measure
MBPP+ improves correctness checking, not task breadth. It does not evaluate real repositories, package selection, code review, multi-file changes, interactive repair, security, style, or asymptotic efficiency except when tests trigger a timeout. It remains Python-only and entry-level by design.
Known issues & criticisms
The largest comparability problem is version drift. EvalPlus v0.2.0 removed broken tasks and later fixed ground-truth solutions for specific IDs, so old and current scores do not share an identical test. Generated tests also inherit ambiguity from underspecified descriptions and dependence on canonical implementations. Running model code without containment is unsafe.
Contamination notes
Risk is high. More tests make accidental false positives less likely but do not make the benchmark private. Both original MBPP material and MBPP+ additions are downloadable. Training on generated tests or canonical solutions can directly optimize the score.
How to run it
Use the current EvalPlus release and isolate execution:
python -m pip install --upgrade evalplus
evalplus.codegen --model gpt-4o-2024-08-06 \
--dataset mbpp --backend openai --greedy
evalplus.evaluate --dataset mbpp --samples path/to/generated_samples.jsonlArchive the installed EvalPlus version or commit, model snapshot, prompt mode, decoding parameters, Python version, and both base and plus results.
History & lineage
MBPP+ was added in EvalPlus v0.2.0 on 2023-11-24, extending the framework beyond HumanEval+. On 2024-04-17, a pre-v0.3.0 update removed broken tasks and changed the set from 399 to 378; later updates corrected several ground-truth solutions. It supersedes original MBPP for evaluator rigor while retaining the same basic task family.
Further reading
- EvalPlus paper and methodology: https://arxiv.org/abs/2305.01210
- Official implementation and release history: https://github.com/evalplus/evalplus
- Official MBPP+ dataset: https://huggingface.co/datasets/evalplus/mbppplus
- Official leaderboard: https://evalplus.github.io/leaderboard.html
- BenchmarkList snapshot used for the five displayed rows: https://benchmarklist.com/benchmarks/mbpp/
- OpenAI o1-preview and o1-mini launch: https://openai.com/index/introducing-openai-o1-preview/
- Qwen2.5-Coder-32B-Instruct launch: https://qwenlm.github.io/blog/qwen2.5-coder-family/
- DeepSeek-Coder-V2 primary paper and release: https://arxiv.org/abs/2406.11931
- Gemini 1.5 Pro 002 launch: https://developers.googleblog.com/en/updated-gemini-models-reduced-15-pro-pricing-increased-rate-limits-and-more/
- Original MBPP paper: https://arxiv.org/abs/2108.07732
- WizardCoder official repository and EvalPlus results: https://github.com/nlpxucan/WizardLM
- DeepSeek-Coder-V2 official repository and result table: https://github.com/deepseek-ai/DeepSeek-Coder-V2
- Qwen2.5-Coder technical report: https://arxiv.org/abs/2409.12186