Overview
Terminal-Bench evaluates whether an agent can complete complex work through a real command line. Terminal-Bench 2.0 contains 89 containerized tasks inspired by software engineering, system administration, security, data processing, machine learning, and scientific workflows. Every task provides an instruction, environment, human-written reference solution, and executable verifier.
Version 2.1 is the current protocol. It keeps the 89-task identity but repairs tasks whose dependencies drifted, resource budgets blocked valid solutions, or instructions disagreed with tests. Because 28 tasks changed, 2.0 and 2.1 scores must be treated as different axes.
Task details
An agent receives a textual objective and terminal access inside a prepared environment. It may inspect files, install or invoke tools subject to policy, write code, build programs, start services, and debug failures. After its time budget, task-specific tests inspect the final container state. Tasks range from building a Linux kernel under QEMU to configuring services, repairing concurrent code, training a model, or resolving a security flaw.
Example
The official site exposes build-linux-kernel-qemu: the agent must build Linux 6.9, generate an initramfs from a provided list, and boot it with QEMU so the verifier can observe the expected system. This is not a code-snippet question; compilation, filesystem state, command choice, and runtime behavior all matter.
Scoring methodology
Each task produces a binary verifier outcome and aggregate accuracy averages those outcomes. The result belongs to an agent-model pair, not a model alone: Codex CLI, Claude Code, Gemini CLI, and Terminus can score differently with the same underlying model. The 2.1 leaderboard requires at least five trials per task for submission, while individual reported rows may summarize those trials differently; preserve trial count and aggregation.
The official leaderboard states that a Terminal-Bench team member ran and verified every result below. Confidence intervals, hack adjustments, and total evaluation costs are reproduced exactly from the supplied 2.1 snapshot.
| Rank | Agent | Model | Effort | Accuracy | Date | Agent org | Model org | PR | Hacks | Cost |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Code | Fable 5 | xhigh | 83.8% ± 1.2% | 2026-06-07 | Anthropic | Anthropic | #75 | -0.2% | $552.67 |
| 2 | Codex | GPT-5.5 | xhigh | 83.1% ± 1.1% | 2026-05-01 | OpenAI | OpenAI | #45 | -0.2% | $2,059.19 |
| 3 | Terminus 2 | Fable 5 | high | 80.4% ± 1.2% | 2026-06-05 | Terminal-Bench | Anthropic | #78 | -0.0% | $438.64 |
| 4 | Cursor CLI | Grok 4.5 | high | 79.3% ± 1.5% | 2026-07-09 | Cursor | xAI | #86 | -9.0% | $134.09 |
| 5 | Claude Code | Opus 4.8 | high | 78.9% ± 1.3% | 2026-07-09 | Anthropic | Anthropic | #92 | -0.0% | $286.94 |
| 6 | Codex | GPT-5.6 Terra | max | 78.4% ± 1.3% | 2026-07-11 | OpenAI | OpenAI | #115 | -0.2% | $421.15 |
| 7 | Terminus 2 | GPT-5.5 | xhigh | 78.0% ± 1.2% | 2026-05-01 | Terminal-Bench | OpenAI | #47 | -0.2% | $493.85 |
| 8 | mini-SWE-agent | Muse Spark 1.1 | xhigh | 76.2% ± 1.2% | 2026-07-09 | Princeton | Meta | #94 | -0.0% | $198.05 |
| 9 | Codex | GPT-5.6 Luna | max | 75.7% ± 1.3% | 2026-07-11 | OpenAI | OpenAI | #112 | -0.9% | $241.45 |
| 10 | Claude Code | Sonnet 5 | high | 74.6% ± 1.6% | 2026-07-09 | Anthropic | Anthropic | #98 | -0.7% | $288.18 |
| 11 | Terminus 2 | Gemini 3 Pro | high | 73.9% ± 1.3% | 2026-05-01 | Terminal-Bench | #48 | -0.5% | $224.44 | |
| 12 | Claude Code | Opus 4.7 | max | 68.9% ± 1.4% | 2026-05-01 | Anthropic | Anthropic | #44 | -0.5% | $599.52 |
| 13 | Terminus 2 | Opus 4.7 | max | 66.1% ± 1.4% | 2026-05-01 | Terminal-Bench | Anthropic | #46 | -0.0% | $582.26 |
| 14 | Gemini CLI | Gemini 3 Pro | high | 65.8% ± 1.4% | 2026-05-01 | #66 | -0.5% | $247.76 | ||
| 14 | Gemini CLI | Gemini 3.1 Pro | high | 65.8% ± 1.7% | 2026-05-05 | #68 | -0.2% | $236.49 | ||
| 16 | Terminus 2 | Gemini 3.1 Pro | high | 65.6% ± 1.7% | 2026-05-05 | Terminal-Bench | #69 | -0.5% | $229.99 | |
| 17 | Claude Code | GLM-5.1 | max | 58.7% ± 1.2% | 2026-05-01 | Anthropic | Z.ai | #67 | -0.0% | $277.14 |
What it doesn’t measure
Terminal-Bench samples 89 curated container tasks, not routine desktop work, GUI use, interpersonal collaboration, production access controls, or long-term ownership. A passing final state does not reveal whether the trajectory was safe, efficient, or maintainable.
Known issues & criticisms
The 2.1 release documents three defects in 2.0: nine tasks had changing external dependencies, eight had resource limits that blocked at least one valid solution, and other tasks had instruction-test mismatches. Changes raised representative agent-model scores by as much as 12.1 points and even lowered two pairs slightly, proving the versions are not interchangeable. Public tests and agent access also create reward-hacking and leaderboard-integrity risks; the maintainers introduced trajectory review and policy updates in 2026.
Contamination notes
Risk is high because task instructions, environments, tests, and solutions are public and static. A canary asks training pipelines not to ingest the data, but compliance is voluntary. Continuous validation in 2.1 detects environmental drift; it does not create fresh hidden tasks.
How to run it
Install Docker and Harbor, authenticate to Harbor Hub if needed, then run 2.1 explicitly:
uv tool install harbor
harbor run \
-d terminal-bench/terminal-bench-2-1 \
-a terminus-2 \
-m anthropic/claude-opus-4-6 \
-k 1 \
-n 4For a leaderboard submission, the official 2.1 repository requires -k 5, public upload, and trajectory review. Reproduce historical 2.0 only with --dataset terminal-bench@2.0; label it 2.0 and never compare it directly with 2.1.
History & lineage
Terminal-Bench 1.0 launched on 2025-05-19. Version 2.0 and the Harbor harness launched on 2025-11-07 with 89 harder, more heavily verified tasks; its paper appeared on 2026-01-17 and was accepted to ICLR 2026. Version 2.1 launched on 2026-05-06, fixing 28 tasks and adding continuous validation. Terminal-Bench 3.0 is in development, so it is not declared as the successor yet.
Further reading
- Terminal-Bench 2.0 paper: https://arxiv.org/abs/2601.11868
- Official 2.0 announcement: https://www.tbench.ai/news/announcement-2-0
- Official 2.1 release, paired results, and defect analysis: https://www.tbench.ai/news/terminal-bench-2-1
- Official 2.1 dataset and submission protocol: https://github.com/harbor-framework/terminal-bench-2-1
- Harbor evaluation framework: https://github.com/harbor-framework/harbor
- Official project and leaderboard: https://www.tbench.ai
- Official Terminal-Bench 2.1 leaderboard used for all 17 rows: https://www.tbench.ai/leaderboard/terminal-bench/2.1