Overview
SWE-bench Verified is a 500-instance subset of the original SWE-bench test set. OpenAI worked with the benchmark authors and 93 Python developers to review 1,699 candidate tasks. Each candidate received three independent annotations for issue clarity, test scope, and estimated difficulty; the final set conservatively excluded any task that one reviewer marked seriously problematic.
Verified became the standard coding-agent headline. It is classified as saturated here because leading mini-SWE-agent 2.0.0 results are tightly clustered—76.8%, 75.8%, 75.8%, and 75.6% resolved—while a later OpenAI audit found residual task-quality problems and evidence that benchmark exposure affected performance. Saturation does not erase the benchmark’s historical importance or the care invested in constructing its 500 engineer-reviewed tasks; it means current score differences carry less clean frontier signal.
Task details
The task and grader match SWE-bench: an agent receives a repository at a base commit and a GitHub issue, uses repository and terminal tools, and submits a patch. Dockerized tests require specified failures to become passing without regressing previously passing tests. The 500 tasks deliberately include as many longer engineer-estimated fixes as possible.
Example
OpenAI’s later audit discusses pylint-dev__pylint-4551: the reference patch adds get_annotation, while alternate plausible implementations can satisfy the visible issue yet fail hidden expectations. This is useful precisely as a cautionary example—passing the test suite and satisfying the natural-language issue are not always identical objectives.
Scoring methodology
The score is pass@1 resolved percentage over 500 tasks. Some OpenAI system cards report a fixed 477-task subset that ran on internal infrastructure; those numbers are not directly comparable to the official 500-task leaderboard and are not plotted here. Model, agent scaffold, tool access, token budget, timeout, and sampling policy must accompany every result.
The supplied leaderboard snapshot uses mini-SWE-agent 2.0.0 throughout. Trajs and Org were blank in the supplied data, so they are not inferred.
| Model | Resolved | Avg. $ | Date | Agent |
|---|---|---|---|---|
| Claude 4.5 Opus (high reasoning) | 76.80% | $0.75 | 2026-02-17 | 2.0.0 |
| Gemini 3 Flash (high reasoning) | 75.80% | $0.36 | 2026-02-17 | 2.0.0 |
| MiniMax M2.5 (high reasoning) | 75.80% | $0.07 | 2026-02-17 | 2.0.0 |
| Claude Opus 4.6 | 75.60% | $0.55 | 2026-02-17 | 2.0.0 |
| GPT-5-2 Codex | 72.80% | $0.45 | 2026-02-19 | 2.0.0 |
| GLM-5 (high reasoning) | 72.80% | $0.53 | 2026-02-17 | 2.0.0 |
| GPT-5-2 (high reasoning) | 72.80% | $0.47 | 2026-02-17 | 2.0.0 |
| GPT 5.2 Codex | 72.80% | $0.45 | 2026-02-19 | 2.0.0 |
| Claude 4.5 Sonnet (high reasoning) | 71.40% | $0.66 | 2026-02-17 | 2.0.0 |
| Kimi K2.5 (high reasoning) | 70.80% | $0.15 | 2026-02-17 | 2.0.0 |
| DeepSeek V3.2 (high reasoning) | 70.00% | $0.45 | 2026-02-17 | 2.0.0 |
| Gemini 3 Pro | 69.60% | $0.96 | 2026-02-26 | 2.0.0 |
| Claude 4.5 Haiku (high reasoning) | 66.60% | $0.33 | 2026-02-17 | 2.0.0 |
| GPT-5 Mini | 56.20% | $0.05 | 2026-02-17 | 2.0.0 |
What it doesn’t measure
Verified is still limited to public historical issues from a small group of Python repositories. It does not measure greenfield work, product judgment, collaboration, deployment, sustained maintenance, or general software engineering across languages and private codebases.
Known issues & criticisms
In 2026 OpenAI audited 138 tasks that o3 did not solve consistently across 64 runs. At least six experienced engineers reviewed each case; 59.4% contained material problems in test design or description. The report identifies overly narrow tests, overly broad tests that accept incomplete solutions, environment-dependent failures, and contamination. The audit sampled persistent failures rather than all 500 tasks, so 59.4% is not an estimate that 59.4% of the whole benchmark is broken—but it is decisive evidence that the remaining error pool is not a clean capability signal.
Contamination notes
Risk is high. All 500 issues and their human solutions are recoverable from public repository history, and the subset itself has been heavily used for training, agent tuning, and regression testing. Human verification improved label quality, not secrecy or freshness.
How to run it
git clone https://github.com/SWE-bench/SWE-bench.git
cd SWE-bench
python -m pip install -e .
python -m swebench.harness.run_evaluation \
--dataset_name SWE-bench/SWE-bench_Verified \
--predictions_path predictions.jsonl \
--max_workers 4 \
--run_id my-verified-runReport whether all 500 tasks ran. Do not label a 477-task internal subset result as the official 500-task score.
History & lineage
Verified launched on 2024-08-13 and superseded the original full and Lite sets for reliable comparison. Its rapid adoption made it the dominant coding-agent benchmark. By the 2026 mini-SWE-agent 2.0.0 snapshot, several leading systems occupied a narrow band near 76% resolved, supporting its saturated classification here. OpenAI separately stopped using it after a deeper audit showed that both contamination and residual grader defects had become material. No single clean successor is declared here: OpenAI briefly recommended SWE-bench Pro, then retracted that recommendation after auditing Pro too.
Further reading
- Construction report, rubric, and launch result: https://openai.com/index/introducing-swe-bench-verified/
- 2026 audit and deprecation rationale: https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/
- Official dataset documentation: https://www.swebench.com/SWE-bench/guides/datasets/
- Official harness: https://github.com/SWE-bench/SWE-bench
- Official leaderboard and standardized mini-SWE-agent results: https://www.swebench.com/