benchwiki
coding

MultiPL-E

Deprecated
2022-08-17 → today

Parallel HumanEval and MBPP translations for comparing code generation across many programming languages.

last updated 2026-08-03

Performance over time

vendor-reported independent
102030405020222023

Baseline: No official human baseline is reported; the benchmark compares model-generated translations of HumanEval or MBPP tasks, not humans taking a calibrated multilingual exam.

Overview

MultiPL-E is both a translation system and a family of multilingual code-generation tests. It takes unit-test-driven Python benchmarks—originally HumanEval and MBPP—and uses small, language-specific translators to rewrite function signatures, tests, types, doctests, and Python terminology for other programming languages. The paper introduced translations into 18 additional languages; the maintained official dataset card now advertises 22 supported languages.

Its main value is parallelism: broadly the same programming problem can be measured across languages, exposing whether a model’s apparent coding ability depends on Python prevalence rather than transferable program synthesis.

Task details

Choose a source set (humaneval or mbpp) and a target-language configuration such as C++, Java, Rust, Racket, or Lua. Each prompt contains a translated natural-language specification and target-language function scaffold. The model completes the implementation; the harness compiles or interprets it and runs translated unit tests in the appropriate language toolchain.

The original paper datasheet counts 2,898 instances: 161 retained HumanEval problems multiplied by 18 translated languages. The current hub also exposes MBPP and later languages, so results must name source set, language, and revision.

Example

A HumanEval prompt asking for a Python function that removes vowels may appear as a typed target-language function with equivalent examples and assertions. A correct completion must use that language’s string and collection conventions and pass the translated tests. Merely emitting the original Python solution is invalid, even if its algorithm is right.

Scoring methodology

The primary metric is pass@1: the fraction of tasks for which one generated program passes every unit test. Multi-sample studies also estimate pass@k. The official repository recommends many samples for reliable pass@10 or pass@100 estimates and warns that a small sample count is suitable only for pass@1.

Scores are language-specific. Compiler availability, timeout policy, prompt terminology, translated doctests, and language-version differences are part of the measurement. A macro-average across languages is useful only alongside the per-language table.

What it doesn’t measure

MultiPL-E measures short function synthesis, not repository maintenance, debugging a running service, dependency selection, code review, or long-horizon agent work. Because the problems originate in HumanEval and MBPP, it also does not create new algorithmic content or eliminate their difficulty ceiling.

Known issues & criticisms

  • Translation equivalence is imperfect. Language type systems and standard libraries differ, so some translations are easier, harder, or less idiomatic than their Python source.
  • Version changes matter. The official dataset changelog records fixes affecting Lua and TypeScript and later additions such as Dart and Ada. Results from different revisions may not be comparable.
  • Language averages conceal failures. Strong Python performance can coexist with weak results in low-resource or structurally different languages.
  • Execution is environment-sensitive. Compiler versions, resource limits, and sandbox configuration can change outcomes.

Contamination notes

Risk is high. Prompts and tests are public, and the underlying HumanEval and MBPP semantics have circulated widely. Translation can reduce verbatim overlap with Python training examples, but a model can still recognize the original problem and reproduce a memorized algorithm. MultiPL-E is most informative for cross-language consistency, not as a clean test of unseen problem solving.

How to run it

The official framework generates completions, executes them in a network-disabled container, and computes pass rates:

git clone https://github.com/nuprl/MultiPL-E
cd MultiPL-E
podman pull ghcr.io/nuprl/multipl-e-evaluation
podman tag ghcr.io/nuprl/multipl-e-evaluation multipl-e-eval
# Generate completions following the repository tutorial, then evaluate them:
podman run --rm --network none -v ./tutorial:/tutorial:rw \
  multipl-e-eval --dir /tutorial --output-dir /tutorial --recursive
python3 evaluation/src/pass_k.py ./tutorial/*

Record the exact dataset revision, source benchmark, target language, prompt variant, temperature, number of samples, and toolchain image.

History & lineage

The 2022 paper introduced MultiPL-E as translations of HumanEval and MBPP into 18 additional languages. The maintained dataset has since added languages and versioned prompt fixes, reaching 22 advertised languages. It remains a descendant of both source benchmarks rather than a harder successor: the tasks stay parallel so researchers can isolate language effects.

Further reading