benchwiki
coding

CodeContests

Deprecated
2022-02-02 → today

DeepMind's competitive-programming corpus with temporally split problems, human submissions, and generated correctness tests.

last updated 2026-08-03

Performance over time

vendor-reported independent
20302022

Baseline: No human 10@100k solve-rate baseline was published. AlphaCode's human comparison was average simulated rank in the top 54.3% across ten Codeforces contests, an incompatible percentile that must not be plotted on the solve-rate axis.

Overview

CodeContests is the dataset behind DeepMind’s AlphaCode. It moves beyond short function completion to full competitive-programming problems requiring algorithm design, input parsing, output formatting, and attention to time and memory constraints. The released corpus contains 13,328 training, 117 validation, and 165 test problems, plus correct and incorrect human submissions in several languages.

The benchmark is now deprecated for frontier comparison. DeepMind constructed a temporal split so validation and test problems postdated the training corpus, but the released problems and solutions no longer provide a clean test for later models; the official repository was archived in 2024.

Task details

Each record includes a problem statement, source and difficulty metadata, example and hidden input/output tests, generated tests, resource limits, and human solutions labeled by outcome. A model writes a complete C++, Python, or Java program. AlphaCode sampled enormous candidate pools, discarded programs that failed examples, clustered candidates by behavior on generated inputs, and submitted a small diverse set.

Example

A typical Codeforces-style task describes an input with many test cases, constraints large enough to rule out brute force, and an exact output format. The model must infer an algorithm, implement it as a complete program, and satisfy hidden edge cases within resource limits. A quadratic solution can be logically correct on small generated tests yet time out on the official judge.

Scoring methodology

The dataset metric associated with AlphaCode is 10@100k: sample 100,000 programs, filter and cluster them, then count a problem solved if one of up to ten selected submissions passes. It is a system metric, not ordinary pass@1. For the paper’s headline human comparison, DeepMind simulated ten Codeforces contests and calculated placements and an estimated rating; AlphaCode averaged the top 54.3% of participants.

What it doesn’t measure

CodeContests measures contest problem solving, not ordinary software engineering. There is no repository context, user requirement negotiation, maintenance, testing strategy, code review, or long-lived dependency environment. Massive sampling and filtering also measure inference-time search infrastructure in addition to the underlying model.

Known issues & criticisms

The AlphaCode paper found a major evaluator gap: generated tests reduced false positives from about 60% to 4%, but 42% of candidate solutions judged functionally correct were still too slow on larger inputs, motivating official-site evaluation. Full hidden contest tests were unavailable. The released repository supports Linux/clang and hard-codes legacy Python paths in places; it was archived on 2024-12-06. A later CodeContests+ paper specifically targets test-quality limitations.

Contamination notes

Risk is high for new evaluations. The original training/evaluation chronology was a strong mitigation: model training material preceded validation and test problems. Once CodeContests, its human solutions, and tests became public, that guarantee applied only to the original AlphaCode experiment. Current models require new contests or private tests for equivalent cleanliness.

How to run it

The official evaluator is Linux-oriented and uses Bazel:

git clone https://github.com/google-deepmind/code_contests.git
cd code_contests
bazel build -c opt :print_names_and_sources
# Download the ~3 GiB Riegeli files as documented in the repository, then:
bazel run -c opt execution:solve_example -- \
  --valid_path=/tmp/dm-code_contests/code_contests_valid.riegeli

Sandbox all candidate programs. Record split, language, compiler/interpreter, resource limits, number of samples, filtering tests, clustering method, and maximum submissions.

History & lineage

DeepMind announced AlphaCode and CodeContests on 2022-02-02; the preprint followed in March and the final paper appeared in Science in December 2022. AlphaCode’s competition simulation established that large-scale sampling plus behavioral filtering could reach a mid-field human rank. AlphaCode 2 later used an updated internal dataset, while the public repository was archived in 2024. CodeContests+ (2025) proposes higher-quality generated tests for the public corpus.

Further reading