benchwiki
agentic-tool-use

Tool Decathlon / Toolathlon

Active
2025-10-29 → today

A long-horizon tool-use benchmark with 108 tasks across 32 software applications, 604 MCP tools, and state-based evaluators.

last updated 2026-08-30

Performance over time

vendor-reported independent
70802026

Baseline: No single protocol-comparable human baseline is published for the full Toolathlon-Verified set.

Overview

Toolathlon, formally the Tool Decathlon, evaluates language agents on realistic long-horizon workflows across many applications. Instead of testing one API or one environment, it combines 32 software applications, 604 MCP tools, and 7 local toolkits into 108 tasks with state-based evaluators.

The benchmark targets a practical gap in agent evaluation: real users often ask agents to coordinate email, calendars, files, spreadsheets, databases, commerce systems, cloud tools, and internal records. Toolathlon asks whether an agent can carry out those workflows reliably rather than merely choose a single function call.

Task details

A task gives the agent an objective and access to a configured tool environment. The agent may need to inspect state, call multiple tools, update records, create artifacts, and verify intermediate results. The paper reports roughly 20 turns on average, and the official leaderboard also reports average turns and tool calls for valid trajectories.

Toolathlon-Verified, released on 2026-06-30, is the current score series. It revised prompts, ground truths, evaluators, initial states, and infrastructure, so older leaderboard scores should be treated as historical.

Example

A representative workflow might ask the agent to check an inbox for assignment submissions, cross-reference a course-management system, grade or summarize the work, update records, and notify the relevant user. Success depends on final application state, not on whether the trajectory looks plausible.

Scoring methodology

The primary metric is Pass@1 over the 108 Toolathlon-Verified tasks. Each task has a dedicated evaluator that checks the resulting state. The official leaderboard also reports Pass@3, Pass^3, average turns, and average tool calls for supported runs.

Because the benchmark is long-horizon, the measured object is an agent system: model, tool schema, planning loop, memory, error recovery, rate-limit behavior, and execution environment. A model-only comparison is incomplete unless the scaffold is held fixed.

The timeline uses model launch dates rather than leaderboard row dates. The paper’s Claude-4.5-Sonnet 38.6% point is not plotted because Claude Sonnet 4.5 launched before Toolathlon’s 2025-10-29 first release.

What it doesn’t measure

Toolathlon does not fully measure security policy adherence, human approval workflows, real production blast radius, interpersonal judgment, or months-long ownership. It also depends on curated application states rather than a user’s live personal data.

Known issues & criticisms

  • Setup can be heavy. Running the full benchmark locally can require Docker or Podman, uv, configured accounts, app-aware credentials, and enough machine resources.
  • Public tasks can be overfit. The tasks and evaluators are available in the repository, which is good for reproducibility but weakens hidden-test integrity.
  • Application realism creates moving parts. Tool schemas, initial states, credentials, and container behavior can all affect scores.
  • Verified reset the axis. The official site explicitly says Toolathlon-Verified results are not directly comparable with the archived pre-Verified leaderboard.

Contamination notes

Risk is high because Toolathlon-Verified is public and static. State-based evaluation reduces grading ambiguity, but it does not prevent memorization of task prompts, expected states, or evaluator logic. Use fresh private tasks or held-out variants for high-stakes claims.

How to run it

The official docs describe several modes. For the public evaluation service:

python eval_client.py run \
  --mode public \
  --base-url YOUR_OPENAI_COMPATIBLE_ENDPOINT \
  --api-key YOUR_API_KEY \
  --model-name YOUR_MODEL_NAME \
  --workers 10 \
  --output-file ./results/eval_stats.json \
  --log-file ./results/client.log \
  --server-log ./results/server.log \
  --traj-log ./results/traj_log_all.jsonl

For local evaluation, use the official repository, install with uv, pull the prepared Toolathlon image, configure credentials, and run the selected finalpool/{task_name} task. Always report whether the run used Toolathlon-Verified, the agent scaffold, and the tool-call budget.

History & lineage

The Tool Decathlon paper was submitted on 2025-10-29 and later marked as ICLR 2026. The first paper reported that Claude-4.5-Sonnet reached 38.6% success while the strongest open-weights model reported there, DeepSeek-V3.2-Exp, reached 20.1%.

Toolathlon-Verified was released on 2026-06-30 as the current benchmark release. It begins a new score series after a major verification pass over task specifications and evaluators.

Further reading