Overview
MBPP—Mostly Basic Programming Problems—contains 974 short Python synthesis tasks intended to be solvable by entry-level programmers. Each item pairs a natural-language description with reference code and usually three assertions. The standard split uses 374 training, 90 validation, 10 prompt/few-shot, and 500 test tasks; the structured size on this page follows that test split.
MBPP helped establish natural-language-to-Python evaluation alongside HumanEval, but its sparse tests and fully public content make it saturated for frontier comparisons. Prefer MBPP+, which retains the useful task distribution while strengthening and cleaning evaluation.
Task details
The model sees a short description such as “write a function to find the shared elements of two lists,” often with a few assertion examples depending on the prompt protocol. It emits a complete Python program or function. The evaluator executes the supplied assertions against that output. Tasks exercise strings, lists, arithmetic, basic algorithms, and standard-library use rather than large systems.
Example
One paper example asks for the minimum cost to connect ropes, with lengths [4, 3, 2, 6], and expects 29. A correct implementation repeatedly combines the two shortest ropes using a priority queue. A program that combines inputs in their given order may pass a convenient example yet fail other orderings, illustrating the need for stronger tests.
Scoring methodology
A task passes only when its generated program executes and satisfies every test. The original paper studied both the fraction of individual samples that succeed and the fraction of tasks solved by at least one among many samples. Those quantities are not interchangeable. Modern reports often use pass@1 on the 500-task test split, sometimes after “sanitizing” prompt/reference overlaps. Always state split, prompt examples, sampling count, and whether the original or sanitized data were used.
Current CodeSOTA pass@1 snapshot
CodeSOTA’s MBPP pass@1 leaderboard is used here as a current registry snapshot. The table keeps CodeSOTA’s rank and score order; the chart dates use model launch dates rather than CodeSOTA’s registry year.
| Rank | Model | Score | Launch date used |
|---|---|---|---|
| 1 | o4-mini | 94.9 | 2025-04-16 |
| 2 | o3-mini | 93.3 | 2025-01-31 |
| 3 | Claude Opus 4 | 92.0 | 2025-05-22 |
| 4 | Claude 3.5 Sonnet (Oct 2024) | 91.0 | 2024-10-22 |
| 5 | GPT-4.1 | 90.9 | 2025-04-14 |
| 6 | Qwen2.5-Coder-32B-Instruct | 90.2 | 2024-11-12 |
| 7 | Qwen2.5-Coder 32B | 90.2 | 2024-11-12 |
| 8 | Claude Sonnet 4 | 89.6 | 2025-05-22 |
| 9 | DeepSeek-Coder-V2-Instruct | 89.4 | 2024-06-14 |
| 10 | DeepSeek-V3 | 89.3 | 2024-12-26 |
What it doesn’t measure
MBPP measures small Python programs from short descriptions. It does not assess repository context, multi-file edits, debugging, unfamiliar APIs, code explanation, security, maintainability, or long-horizon algorithm design. Three examples cannot fully specify behavior, and a passing program may be brittle beyond tested inputs.
Known issues & criticisms
The original paper notes that programs can overfit assertion examples. EvalPlus later expanded MBPP testing and removed broken tasks: MBPP+ v0.2.0 reduced its evaluated set from 399 to 378, with roughly a four-point expected score shift. The original corpus mixes crowd-sourced items with a smaller edited and hand-verified subset, so task quality is uneven. Protocol fragmentation between full, test, prompt, sanitized, and plus variants further complicates comparisons.
Contamination notes
Risk is high for present-day models. The authors’ overlap study supported cleanliness relative to their 2021 model’s pretraining data; it does not establish cleanliness after years of public distribution. Prompts, assertions, and solutions now appear in dataset mirrors and evaluation repositories.
How to run it
Load the official dataset and evaluate generated code inside an isolated environment:
python -m pip install datasets evalplus
python -c "from datasets import load_dataset; print(load_dataset('google-research-datasets/mbpp'))"
evalplus.codegen --model gpt-4o-2024-08-06 --dataset mbpp --backend openai --greedy
evalplus.evaluate --dataset mbpp --samples path/to/generated_samples.jsonlEvalPlus reports both base and stronger plus-test results. For a historical MBPP reproduction, use only the original assertions and reproduce the paper’s prompt and sampling protocol.
History & lineage
Google Research released MBPP with its paper on 2021-08-16. The study evaluated models from 244M to 137B parameters and reported 59.6% of tasks solved by the largest model under a many-sample few-shot setup. Later harnesses standardized a 500-task test split and produced sanitized variants. EvalPlus released MBPP+ in 2023 and removed broken tasks in 2024.
Further reading
- Original paper: https://arxiv.org/abs/2108.07732
- Google Research publication page: https://research.google/pubs/program-synthesis-with-large-language-models/
- Official Google Research data: https://github.com/google-research/google-research/tree/master/mbpp
- Official dataset card: https://huggingface.co/datasets/google-research-datasets/mbpp
- EvalPlus implementation and MBPP+ release notes: https://github.com/evalplus/evalplus
- CodeSOTA MBPP pass@1 leaderboard: https://www.codesota.com/benchmark/mbpp
- OpenAI, “Introducing OpenAI o3 and o4-mini”: https://openai.com/index/introducing-o3-and-o4-mini/
- OpenAI, “OpenAI o3-mini”: https://openai.com/index/openai-o3-mini/
- OpenAI, “Introducing GPT-4.1 in the API”: https://openai.com/index/gpt-4-1/
- Anthropic, “Introducing Claude 4”: https://www.anthropic.com/news/claude-4
- Anthropic Claude Sonnet model announcements: https://www.anthropic.com/claude/sonnet
- Qwen2.5-Coder technical report and launch post: https://qwenlm.github.io/blog/qwen2.5-coder-family/
- DeepSeek API changelog: https://api-docs.deepseek.com/updates
- DeepSeek-V3 official report and results: https://github.com/deepseek-ai/DeepSeek-V3