benchwiki
agentic-tool-use

AgentBench

Deprecated
2023-08-07 → today

A multi-environment benchmark for LLM agents across OS, database, knowledge-graph, game, shopping, household, and web tasks.

last updated 2026-08-31

Performance over time

vendor-reported independent
01020304020232024

Baseline: The benchmark is built from human-solvable interactive tasks, but no single blind human aggregate score is protocol-comparable across all eight original environments.

Overview

AgentBench evaluates LLMs as agents rather than as single-turn text generators. It places a model-backed agent inside interactive task environments and measures whether it can reason over observations, choose actions, use tools, and complete goals over multiple turns.

The original benchmark covers eight environments: operating system interaction, database interaction, knowledge-graph querying, a digital card game, lateral-thinking puzzles, ALFWorld household tasks, WebShop shopping tasks, and Mind2Web web-browsing tasks. The current repository also documents AgentBench FC, a function-calling version that containerizes five core environments and connects them to AgentRL.

Task details

Each task runs through a task server, an agent server, and a client that forwards observations and actions. The agent receives a goal or instruction, produces an action in the environment’s required format, receives feedback, and continues until it succeeds, fails, or reaches the configured limit.

The environments intentionally vary. OS tasks can require shell operations, database tasks require SQL interactions, knowledge-graph tasks require querying a large graph, and shopping or household tasks require a sequence of actions in a simulated interface. This diversity is the main value of AgentBench: it makes brittle single-environment agent claims harder to hide.

Example

A database task may ask for an answer that can only be obtained by inspecting a schema, forming SQL, executing the query, reading the result, and deciding whether another query is needed. An OS task may instead ask the agent to inspect a live filesystem and modify file permissions or extract deterministic information from command output.

Scoring methodology

The headline metric is the overall AgentBench score (OA) from Table 3: a weighted average of 8 environments, reported in the paper as OA out of 10 (max 10). This site normalizes to percent (OA×10) for chart parity — e.g. GPT-4 0613 OA 4.01 → 40.1%, Claude 3 Opus 3.11 → 31.1%, GLM-4 2.89 → 28.9%, CodeLlama-34B 0.96 → 9.6% (https://arxiv.org/html/2308.03688v3#S4.T3). Reported overall scores should state the benchmark version, split, task subset, model, agent scaffold, prompt format, maximum turns, and tool permissions.

Scores from original AgentBench and AgentBench FC should not be placed on the same axis unless the environment set, prompt style, and scorer are identical. FC changes the interface from free-form action text to function calling and changes deployment assumptions.

The timeline uses model launch dates per this site’s graph convention for this update. GPT-4 0613 (OA 4.01 → 40.1%) and GPT-3.5-turbo 0613 (2.32 → 23.2%) are intentionally not plotted because their model launch dates precede AgentBench’s 2023-08-07 first release.

What it doesn’t measure

AgentBench does not measure long-running production autonomy, human handoff, safety under real credentials, GUI desktop work, or open-ended project ownership. It is a suite of bounded interactive environments, so a high score is evidence of controlled task execution rather than general reliability.

Known issues & criticisms

  • Environment heterogeneity complicates aggregation. OS, SQL, web, game, and puzzle tasks fail for different reasons, so the mean score can hide a narrow strength or weakness.
  • Scaffolding matters. Memory, action parsing, retry policy, and observation formatting are part of the evaluated system.
  • Reproducibility is operationally heavy. Some environments require Docker images, substantial RAM, or external services such as a local Freebase/SPARQL setup.
  • Version drift matters. The 2025 AgentBench FC update is useful, but it is a new protocol and should be reported separately from original AgentBench v0.1/v0.2 results.

Contamination notes

Risk is medium. The framework, many environment descriptions, and dev materials are public. The official test split is not fully public, which helps leaderboard integrity, but agents can still overfit to the visible task formats, prompts, and environment APIs.

How to run it

For the current function-calling/containerized setup, start from the official repository:

git clone https://github.com/THUDM/AgentBench.git
cd AgentBench
docker compose -f extra/docker-compose.yml up

Then configure the model endpoint and run the repository’s assignment or AgentRL evaluation flow for the selected tasks. For historical comparisons, pin the repository tag or commit and clearly label whether the run uses original AgentBench or AgentBench FC.

History & lineage

AgentBench was submitted to arXiv on 2023-08-07 and published at ICLR 2024. It helped define the “LLM as agent” benchmark pattern: run a model in an interactive environment, score successful completion, and analyze failures in long-term reasoning, decision-making, and instruction following.

The repository later added VisualAgentBench in 2024 for visual foundation agents and AgentBench FC in 2025 for function-calling agents. Those are related variants, not interchangeable score series.

Further reading