SWE-bench Pro

Overview

SWE-bench Pro expanded repository repair to longer, more diverse work. Its 1,865 tasks span 41 repositories: 731 public tasks from 11 open-source repositories, 858 held-out tasks from 12 open-source repositories, and 276 commercial tasks from 18 proprietary startup repositories. Reference patches average 107.4 changed lines across 4.1 files, and experts augmented ambiguous issue text instead of discarding every realistic imperfection.

The benchmark remains active here because its public leaderboard continues to add models and the 2026-08-03 snapshot still spans 0.584 to 0.800 among the 17 supplied leaders. That classification does not erase later quality auditing: on 2026-07-08, OpenAI reported that its pipeline marked 200 public tasks broken and human reviewers marked 249, estimating roughly 30% of the public set was broken. Current scores should therefore be read as an actively tracked but qualification-heavy signal.

Task details

As in SWE-bench, an agent receives a codebase snapshot and an issue-like specification, works through a terminal, and submits a patch. Hidden tests assess the requested feature and regressions. The public split is the reproducible headline configuration; the held-out split is not released, and the commercial split exposes aggregate results without code or tasks.

Example

OpenAI’s audit gives a concrete public task, OpenLibrary-77c16d5. The prompt’s Markdown examples require one leading space, while hidden assertions require two. An agent that follows the visible specification exactly fails. This illustrates why a unit-test grader is only valid when prompt and tests define the same contract.

Scoring methodology

The primary score is public-split pass@1: the share of 731 tasks whose generated patch passes the evaluation suite. Launch comparisons used a common SWE-agent scaffold. Public and commercial scores are separate axes and must not be averaged. Dataset test removals, timeout fixes, uncapped reruns, and leaderboard corrections mean later rows require an exact dataset and harness revision.

The following is the complete user-supplied top-17 snapshot from LLM Stats, last updated 2026-08-03. Raw 0–1 scores are preserved here; the performance timeline multiplies them by 100 to match this page’s percentage axis. LLM Stats labels its SWE-bench Pro results self-reported and unverified. A dash means the source displayed no value; costs are the source’s listed input/output model prices, not the cost of completing the benchmark.

What it doesn’t measure

Passing Pro’s hidden tests does not establish maintainability, security, code-review quality, deployment success, or correct behavior outside test coverage. The private commercial split broadens code provenance but is not independently auditable.

Known issues & criticisms

OpenAI’s July 2026 audit found four central failure modes: overly strict tests that enforce unstated implementations, underspecified prompts, low-coverage tests that accept incomplete work, and misleading prompts that contradict tests. Its pipeline labeled 27.4% of the public split broken; five-engineer review labeled 34.1%. Separately, the official repository notes removed outdated tests and, in May 2026, warned that leaderboard issues were under investigation. Those changes make the headline trajectory partly a moving measurement instrument.

Contamination notes

Risk is medium overall but split-dependent. Commercial tasks have the strongest secrecy. Held-out tasks come from public repositories but are not released as benchmark items. Public tasks and gold changes are inspectable. Strong-copyleft licensing is a legal deterrent to some training uses, not evidence that a model has never seen the code or pull request.

How to run it

git clone --recurse-submodules https://github.com/scaleapi/SWE-bench_Pro-os.git
cd SWE-bench_Pro-os
python -m pip install -r requirements.txt
python swe_bench_pro_eval.py \
  --raw_sample_path=swe_bench_pro_full.csv \
  --patch_path=predictions.json \
  --output_dir=results \
  --scripts_dir=run_scripts \
  --num_workers=4 \
  --dockerhub_username=jefzda

Generate patches with the bundled SWE-agent or another scaffold, but record the exact repository commit and dataset revision. Do not compare scores produced before and after task removals as if only the model changed.

History & lineage

Scale announced Pro on 2025-09-19; the paper followed on arXiv on 2025-09-21. Launch models remained below 25% on the public split. During 2026 the maintainers removed outdated tests, fixed slow instances, and flagged leaderboard issues. OpenAI’s 2026-07-08 audit then estimated roughly 30% broken tasks and withdrew its recommendation. The benchmark remains active on this page because model reporting continues and the current leaderboard retains meaningful separation, with no declared successor.

Further reading

• Original paper: https://arxiv.org/abs/2509.16941
• Official project description and split counts: https://labs.scale.com/papers/swe_bench_pro
• Official repository, run instructions, and change notices: https://github.com/scaleapi/SWE-bench_Pro-os
• OpenAI’s July 2026 task-quality audit: https://openai.com/index/separating-signal-from-noise-coding-evaluations/
• Official public leaderboard: https://labs.scale.com/leaderboard/swe_bench_pro_public
• LLM Stats 2026-08-03 leaderboard snapshot used for the 17 supplied rows: https://llm-stats.com/benchmarks/swe-bench-pro