benchwiki
coding

SWE-Lancer

Deprecated
2025-02-17 → today

Current 198-task offline patch benchmark derived from paid Upwork work, with historical mixed implementation and manager-task variants.

last updated 2026-08-03

Performance over time

vendor-reported independent
01020302025

Baseline: No blind protocol-comparable aggregate human baseline is reported. Original freelancers completed the IC work and hired managers chose proposals, defining the references rather than a timed test score.

Overview

SWE-Lancer grounds software-agent evaluation in paid freelance work. The full private collection contains 1,488 Upwork tasks worth exactly $1 million in historical payouts. It combines 764 individual-contributor implementation tasks with 724 manager tasks that ask the model to choose among real technical proposals. The public Diamond split contains 502 tasks worth $500,800.

The paper-era pairing measured both changing code and selecting an engineering approach. The maintained public leaderboard now narrows the executable protocol to a 198-task Diamond offline subset. Structured fields on this page follow that current subset; the original 502-task mixed Diamond score is historical and not directly comparable. Historical payout remains a second axis that weights solved tasks by what clients paid.

Task details

The current Diamond offline leaderboard has 198 IC software-engineering tasks. Agents inspect and edit a snapshot of the Expensify codebase with terminal tools; end-to-end tests grade each patch. In the original paper protocol, Diamond instead had 237 IC tasks worth $236,300 plus 265 manager tasks worth $264,500. Manager tasks presented competing implementation proposals and used the historical project choice as the label.

Example

The paper includes manager tasks such as choosing how to correct country-dependent postcode validation. Candidate proposals may all sound plausible, but only the option selected in the historical project receives credit. IC tasks range from small bug fixes to feature work and return a patch rather than a multiple-choice answer.

Scoring methodology

Current leaderboard accuracy micro-averages binary end-to-end test outcomes across the 198 offline patch tasks. Earned payout sums the historical values attached to solved tasks, so accuracy and payout rankings can differ. The original 502-task Diamond overall score additionally mixed IC results with manager-choice accuracy; it must be labeled as the paper protocol. IC test suites were written and triple-verified by experienced engineers.

What it doesn’t measure

Historical payout is not a controlled measure of task difficulty or business value. The benchmark centers one codebase and freelance workflow, and manager choice captures proposal selection rather than planning, hiring, communication, or project delivery.

Known issues & criticisms

IC and manager tasks use fundamentally different graders, so one aggregate can conceal opposite trends. Manager labels reproduce a historical decision, not necessarily the unique best decision. The initial runtime depended on Internet access; OpenAI’s July 2025 update removed that dependency and the maintained leaderboard now covers 198 offline tasks. This is a protocol change, not a drop-in rerun of the paper’s 502-task mixed Diamond set.

Contamination notes

Risk is medium for Diamond. The public code, task data, and evaluator can enter training or agent-tuning corpora, and the collection is static. The unreleased full tasks have lower direct exposure. Offline execution removes temporal network drift but does not mitigate memorization.

How to run it

Use the maintained Frontier Evals repository rather than the archived original repository:

git clone https://github.com/openai/preparedness.git frontier-evals
cd frontier-evals/project/swelancer
uv sync
# Follow this directory's README to prepare the unified image,
# generate model outputs, and run the IC and manager evaluators.

Pin the repository and data revision. For current runs report accuracy and earned payout over all 198 offline tasks; for historical paper runs separately report IC, manager, overall, and payout metrics and label the 502-task protocol.

History & lineage

The paper and original public Diamond split appeared on 2025-02-17. The full benchmark comprises 1,488 tasks; paper-era Diamond exposed 502 mixed IC and manager tasks. On 2025-07-18 the original repository was archived and the code moved into OpenAI’s Frontier Evals repository. The 2025-07-28 update published revised offline results; the maintained public leaderboard evaluates 198 IC patch tasks.

Further reading