benchwiki
coding

SWE-bench

Deprecated
2023-10-10 → today

Real GitHub issues from 12 Python repositories, solved by generating patches that must pass repository tests.

last updated 2026-08-03

Performance over time

vendor-reported independent
0102020232024

Baseline: No protocol-comparable aggregate human score was published. Every instance has a human-authored pull request, but that does not establish a timed blind-test pass rate.

Overview

SWE-bench turned software repair into an executable language-model evaluation. Each of its 2,294 instances pairs a real issue with a historical snapshot of one of 12 popular Python repositories. A system must understand the issue, navigate a repository, and return a patch that satisfies tests. This was a major step beyond function-completion benchmarks because success can require coordinated changes across several files and a working development environment.

The original full set is now best treated as historical. OpenAI and the SWE-bench authors found that some tasks were underspecified or used tests that rejected otherwise valid solutions, then released the human-filtered SWE-bench Verified subset as its replacement.

Task details

An instance provides a repository identifier, base commit, issue text, and metadata. The system outputs a Git diff. During grading, the harness checks out the base commit in a Docker image, applies the patch and test patch, then runs two groups of tests: FAIL_TO_PASS tests must change from failing to passing, while PASS_TO_PASS tests must continue to pass.

Example

A typical task says that a library behaves incorrectly for a particular input and links that description to the pre-fix repository state. A successful agent searches the code and tests, edits the relevant implementation, and emits a minimal diff. The gold pull request is not shown to the agent. Real task fields and example instance IDs are documented in the official dataset guide.

Scoring methodology

The headline result is the percentage of instances resolved. There is no partial credit: a plausible patch scores zero if one target test still fails or a regression appears. Scores therefore measure the whole system—model, repository retrieval, agent scaffold, tool permissions, timeout, and environment—not the base model alone. Report all of those settings with a score.

What it doesn’t measure

SWE-bench covers historical issues from 12 open-source Python repositories. It does not represent greenfield development, code review quality, requirements gathering, collaboration, deployment, non-Python ecosystems, or the maintenance burden of a patch after the tests pass.

Known issues & criticisms

The SWE-bench Verified annotation campaign found problem statements that were underspecified and FAIL_TO_PASS tests that filtered out valid solutions. That makes failures on the unfiltered full set ambiguous. Scores are also highly scaffold-dependent, and public tests can be gamed without producing a generally correct fix.

Contamination notes

Risk is high. The issue text, merged pull request, gold patch, and much of the test history are public. A historical base commit prevents simply applying the current repository state, but it does not prevent memorizing or retrieving the human solution. The static dataset has no refresh mechanism.

How to run it

Use Docker and the official harness:

git clone https://github.com/SWE-bench/SWE-bench.git
cd SWE-bench
python -m pip install -e .
python -m swebench.harness.run_evaluation \
  --dataset_name SWE-bench/SWE-bench \
  --predictions_path predictions.jsonl \
  --max_workers 4 \
  --run_id my-full-run

Pin the dataset revision, harness commit, image architecture, timeout, agent, and model. For new comparisons, prefer Verified only with its current caveats or a newer purpose-built evaluation.

History & lineage

The paper appeared on 2023-10-10 and was accepted as an ICLR 2024 oral. SWE-bench Lite later offered a smaller set; containerized evaluation improved reproducibility in June 2024; Verified arrived on 2024-08-13 after professional-developer review. Multimodal and multilingual variants broadened repository and input coverage but are separate score axes.

Further reading