benchwiki
coding

LiveCodeBench

Saturated
2024-03-12 → today

Continuously updated contest problems for time-windowed code generation, execution, test prediction, and self-repair evaluation.

last updated 2026-08-03

Performance over time

vendor-reported independent
30405060708090100202420252026

Baseline: No official aggregate human baseline is reported. Problems come from human programming contests with platform-specific participant statistics, which are not one comparable LiveCodeBench baseline.

Overview

LiveCodeBench is a rolling coding evaluation built from newly published problems on LeetCode, AtCoder, and Codeforces. Its key design is temporal: every problem has a publication date, so evaluators can choose a window after a model’s training cutoff instead of treating one permanently public test set as clean forever.

It is also broader than a single code-completion score. The project supports code generation, code execution, test-output prediction, and self-repair. The August 2, 2026 leaderboard snapshot is saturated at the top: Qwen3.7 Max scores 91.6%, Qwen3.7 Plus 89.6%, and GLM-4.7 84.9%.

Task details

For code generation, the model receives a contest statement, constraints, and input/output format, then produces a complete program that hidden tests execute. Other scenarios ask the model to predict program outputs, reason about execution, or repair an initially failing generated solution.

The repository versions the corpus: release v1 has 400 problems through 2024-03; v2 511; v3 612; v4 713; v5 880; and v6 1,055 through 2025-04. This page records v6 because it is the latest size explicitly documented by the official README; always specify the release or date range in results.

Example

A task may be a recently released contest problem describing arrays, constraints, and stdin/stdout examples. The model must emit a whole program, not merely a function. A plausible algorithm that mishandles one edge case, exceeds the time limit, or prints extra prose fails the hidden tests.

Scoring methodology

Code-generation results use pass@1 and pass@5 under unit tests. The official runner commonly samples n=10 at temperature 0.2 and computes the estimators, while leaderboard comparisons can select specific dates. Its default code_generation_lite prunes tests for speed; --not_fast selects the original test collection.

Those choices are material. Report the release, start/end dates, lite/full evaluator, sample count, temperature, timeout, and whether the result is pass@1 or pass@5.

What it doesn’t measure

LiveCodeBench primarily measures contest-style algorithmic coding. It does not reproduce large-repository navigation, dependency upgrades, stakeholder clarification, code review, deployment, or weeks-long maintenance. Its holistic scenarios broaden short-program evaluation but remain bounded tasks.

Known issues & criticisms

  • Freshness is relative. A task is clean only relative to a named model cutoff and evaluation date.
  • Release drift breaks comparisons. A bare “LiveCodeBench score” may refer to different releases or time windows.
  • Test-set errata exist. The official repository maintains an ERRATA file for erroneous tests and problems unsuitable for autograding.
  • Timeout sensitivity. The maintainers document small score variation from parallel execution and time limits.
  • Lite versus full tests. The faster default and original evaluator should not be silently mixed.

Contamination notes

Risk is medium. Continuous collection and date filters are a stronger defense than static HumanEval or MBPP, but all released problems are public contest material and eventually enter web and training corpora. “Contamination-free” therefore means a correctly chosen post-cutoff window, not an everlasting property of the full dataset.

How to run it

git clone https://github.com/LiveCodeBench/LiveCodeBench.git
cd LiveCodeBench
uv venv --python 3.11
source .venv/bin/activate
uv pip install -e .
python -m lcb_runner.runner.main --model <model_name> \
  --scenario codegeneration --evaluate --release_version release_v6

For a cutoff-aligned score, use the saved evaluation file with compute_scores.py --start_date YYYY-MM-DD --end_date YYYY-MM-DD. Run generated code only inside the provided isolation setup.

History & lineage

LiveCodeBench launched in March 2024 with 400 problems dated from 2023-05 onward. Successive releases expanded the corpus and the project added fine-grained release windows. It responds directly to the exposure and narrowness of HumanEval-style static function tests while retaining deterministic execution-based grading.

Further reading