benchwiki
agentic-tool-use

BrowseComp

Nearing saturation
2025-04-10 → today

OpenAI's benchmark for browsing agents that must find hard-to-locate, short-answer facts on the web.

last updated 2026-08-31

Performance over time

vendor-reported independent
2030405060702025human baseline 29.2%

Baseline: OpenAI's verification campaign reports trainers solved 29.2% of 1,255 checked problems within the allowed search process; among solved problems, trainer and reference answers matched 86.4%.

Overview

BrowseComp measures whether a browsing agent can find hard-to-locate facts on the open web. The tasks are intentionally “inverted”: the answer is short and easy to verify after the fact, but difficult to discover without persistent search and multi-hop evidence gathering.

OpenAI positioned BrowseComp as a successor-style stress test after simpler factual benchmarks such as SimpleQA became too easy for browsing-enabled frontier models. It is not a benchmark of writing long research reports. It isolates the narrower but important skill of finding a single obscure answer.

Task details

Each item asks for one short, stable, factual answer. The model or agent may browse the web, search repeatedly, inspect candidate sources, and reason over constraints until it can provide the answer. The construction process asked trainers to prefer facts with high verification asymmetry: hard to find, but easy to check once found.

The public OpenAI article gives a small number of examples, but the benchmark maintainers request that users avoid redistributing examples or answers in plaintext to reduce leakage.

Example

An item can describe a target entity indirectly through several constraints: an obscure publication window, a biographical fact, a role, a location, or a property that is not likely to appear in one search result. A successful agent forms search hypotheses, checks sources, eliminates false positives, and returns only the target answer.

Scoring methodology

BrowseComp is scored as short-answer accuracy. The official simple-evals implementation compares the model’s final answer against the reference answer. Because the answers are short, the benchmark avoids the hardest part of grading open-ended research responses, but it also narrows what the score means.

For a meaningful result, report the browsing tools, search engine, maximum wall-clock time, maximum tool calls, whether parallel search or self-consistency was used, and whether the agent could inspect arbitrary web pages.

The timeline uses model launch dates rather than measurement dates. OpenAI’s original BrowseComp table includes GPT-4o, GPT-4.5, o1, and Deep Research, but those launches predate BrowseComp’s 2025-04-10 release, so they are described in prose but not plotted under this date rule.

What it doesn’t measure

BrowseComp does not measure long-form synthesis, citation quality, user-facing report writing, source trust calibration across ambiguous claims, or answers that legitimately change over time. It also does not represent common web-search traffic; the questions are deliberately obscure.

Known issues & criticisms

  • Public static data is fragile. Once tasks and answers are exposed, browsing agents can retrieve benchmark artifacts rather than solve the search problem.
  • Short answers simplify grading but narrow validity. Real browsing tasks often require multi-paragraph synthesis, uncertainty handling, and source comparison.
  • Tooling dominates. Search API quality, browser access, rate limits, and page-fetch reliability can change the score independently of the base model.
  • Human baseline is not a universal ceiling. The reported human campaign used a specific group, time limit, and no AI assistant; it is useful context, not a claim about expert researchers.

Contamination notes

Risk is high because the benchmark is public and static. OpenAI includes a canary string and asks users not to share examples, but those mitigations depend on training-pipeline compliance and community behavior. Treat public BrowseComp scores as useful but vulnerable.

How to run it

Use the BrowseComp evaluator in OpenAI’s simple-evals repository:

git clone https://github.com/openai/simple-evals.git
cd simple-evals
python browsecomp_eval.py

For comparable runs, pin the repository commit and document browser/search access, tool-call limits, timeout, model, agent scaffold, and answer-normalization settings.

History & lineage

BrowseComp was announced by OpenAI on 2025-04-10, and the arXiv paper was submitted on 2025-04-16. It follows the same short-answer factuality philosophy as SimpleQA but makes discovery much harder by requiring persistent browsing.

The simple-evals repository later marked itself as deprecated for new model result updates, but it remains the reference implementation for BrowseComp, SimpleQA, and HealthBench.

Further reading