benchwiki
coding

Mercury

Deprecated
2024-02-12 → today

Python code-synthesis benchmark that rewards both functional correctness and runtime efficiency against real solution distributions.

last updated 2026-08-03

Performance over time

vendor-reported independent
35455520232024

Baseline: No official human Beyond@1 baseline is reported. The comparison reference is a distribution of accepted human-written LeetCode solutions, not people answering the benchmark under controlled conditions.

Overview

Mercury is the code-efficiency benchmark introduced by Du et al., not the later diffusion-based product named Mercury Coder. It asks a question that pass/fail coding tests ignore: when two generated programs are both correct, does one waste far more time than competent existing solutions?

The full corpus contains 1,889 Python tasks derived from LeetCode. The paper selects a difficulty-balanced 256-task Mercury-eval split as its benchmark and uses the remainder for training experiments. Each evaluation task includes a specification, a test-case generator, an entry point, and a distribution of accepted solutions with different runtimes.

Task details

The model receives a structured programming problem and generates a Python solution. Correctness is checked with generated tests. If the program is correct, its runtime is compared with the task’s reference runtime distribution; the benchmark therefore needs both sandboxed execution and stable timing conditions.

The page’s dataset.size is 256 because that is the evaluation split used for reported benchmark results, not the 1,889-task train-plus-eval corpus.

Example

Suppose two submissions solve the same array problem. One uses a linear scan; the other repeatedly sorts or performs nested work. Both may pass every correctness test. Ordinary pass@1 awards them equally, while Mercury places the faster program at a higher percentile of accepted reference solutions and awards it more Beyond credit.

Scoring methodology

Functional correctness is a prerequisite. Pass@1 records whether one generated solution passes the tests. Beyond@k weights passing solutions by the proportion of reference solutions whose runtime they exceed. Paper Table 3 samples five solutions per task; this page chart therefore labels that five-sample protocol rather than calling it Beyond@1. A wrong or non-running solution receives zero.

This combination avoids declaring a very fast wrong answer efficient, but it makes infrastructure part of the benchmark. CPU scheduling, interpreter version, warm-up, repeated timing, and timeouts can move a solution among percentiles. Published comparisons must share one controlled runtime environment.

What it doesn’t measure

Mercury does not measure memory consumption, energy use, readability, maintainability, repository integration, security, or end-to-end developer productivity. It evaluates Python solutions to algorithmic problems; faster benchmark code does not necessarily imply better production software.

Known issues & criticisms

  • Runtime is noisy. Small latency differences can reflect hardware and system load rather than algorithm quality.
  • Reference distributions are platform-bound. Historical accepted solutions and online runtimes are not a universal measure of optimality.
  • Public-problem exposure. The underlying problems and many solutions originated on LeetCode.
  • Generated-test dependence. The paper uses GPT-4-assisted test-generator drafting followed by online validation; generator coverage still defines which programs count as correct.
  • Composite scores are less intuitive. Two systems can reach similar Beyond scores through different correctness/efficiency tradeoffs.

Contamination notes

Risk is high because the source tasks and accepted solutions come from a public coding platform and the released split is static. Solution deduplication helps build the reference distribution but does not prove that a model has not seen the problem or an efficient solution. Mercury is strongest as a controlled efficiency comparison, not a clean novelty test.

How to run it

Use the official repository and follow its environment-specific generation and execution scripts:

git clone https://github.com/Elfsong/Mercury
cd Mercury
# Follow the repository README to install the pinned environment,
# generate model completions, and execute Mercury-eval in its sandbox.

Do not execute untrusted model code outside a sandbox. Report hardware, Python version, timeout, number of timing repetitions, model sampling settings, and both Pass@1 and Beyond@1.

History & lineage

Mercury was released in February 2024 and accepted to the NeurIPS 2024 Datasets and Benchmarks Track. It extends the unit-test code-synthesis tradition of HumanEval, MBPP, and APPS by adding a runtime distribution and the Beyond@k metric. The paper also uses Mercury-train to compare supervised fine-tuning with direct preference optimization for efficient code generation.

Further reading