benchwiki
coding

FrontierCode

Active
2026-06-08 → today

Private maintainer-authored repository tasks graded for mergeability: correctness, tests, scope discipline, style, and code quality.

last updated 2026-08-03

Performance over time

vendor-reported independent
010203040502026

Baseline: No controlled human aggregate is published. Repository maintainers author the tasks and define mergeability criteria, but that provenance is not a measured human score.

Overview

FrontierCode is Cognition’s private coding-agent benchmark for a stricter question than “does the patch pass tests?”: would the repository maintainer merge it? More than 20 open-source developers created tasks from repositories they maintain, and maintainers define criteria covering behavioral correctness, regressions, mechanical cleanliness, tests, scope, and code quality.

This page deliberately identifies the requested FrontierCode with Cognition’s benchmark launched on 2026-06-08. It is unrelated to generic uses of “frontier code models.”

Task details

An agent receives a checked-out repository, a concise maintainer-written task description, and codebase guidelines. It explores the repository, edits code and tests, runs tools, and submits a patch without human intervention. Extended contains all 150 tasks; Main contains the hardest 100. Version 1.1 no longer reports the original 50-task Diamond subset.

Tasks span 36 open-source repositories and multiple languages. They can require internet documentation, so the protocol permits normal search while forbidding access to upstream pull requests, patches, mirrors, or packages that reveal the solution.

Example

An official task may ask an agent to fix a framework bug while preserving neighboring behavior and adding an effective regression test. Passing the hidden behavioral test is necessary but insufficient: an over-broad diff, a test that also passes on the buggy base commit, broken lint, or an unidiomatic workaround can prevent the patch from being considered mergeable.

Scoring methodology

Each task has weighted reviewer-defined criteria. Blockers represent hard stops in review; if any blocker fails, the trial scores zero. If all blockers pass, the trial receives the weighted aggregate of satisfied blocker and non-blocker rubric items. The separate pass rate counts trials clearing every blocker.

Graders include injected unit tests, build/lint commands, “reverse-classical” tests that verify the agent’s new test fails on the base commit, scope checks, adaptive tests, and prompt-based grading for soft qualities. Because prompt graders are involved but the benchmark does not publicly name one fixed judge model, judge_model is null and exact hosted grading remains part of Cognition’s protocol.

What it doesn’t measure

FrontierCode does not cover private enterprise codebases, production deployment, team communication, product judgment, or long-term maintenance after merge. It measures autonomous patch quality on selected open-source repositories under Cognition’s hosted rubric, not all software-engineering work.

Known issues & criticisms

  • Private tasks and grader. External researchers cannot fully inspect or independently reproduce the benchmark.
  • Subjective criteria. Style and code-quality checks encode maintainer preferences and sometimes use prompt-based grading.
  • Best-effort selection. Reporting a model’s best reasoning level after five trials differs from pass@1 evaluation.
  • Revision instability. Version 1.1 demoted 75 overly strict blockers and deprecated Diamond, so 1.0 and 1.1 numbers are not one timeline.
  • Publisher conflict. Cognition creates the benchmark and evaluates coding agents, including its own systems; results should be labeled hosted/vendor-reported unless independently reproduced.

Contamination notes

Risk is low relative to public issue benchmarks because task materials and canonical solutions are withheld. The remaining concern is agent internet use: some tasks derive from real pull-request work. Version 1.1 distinguishes legitimate documentation lookup from solution-bearing access, provides explicit instructions, and zeroes runs caught by a verifier. The mechanism lowers risk but is not independently auditable from public artifacts.

How to run it

There is no public download-and-run harness. Model creators request or use Cognition’s hosted evaluation through the official leaderboard:

https://cognition.com/frontiercode

Report benchmark version, Main or Extended, score versus pass rate, reasoning effort, number of trials, agent scaffold, internet policy, and whether Cognition or an independent party ran the evaluation.

History & lineage

FrontierCode 1.0 launched on 2026-06-08 with nested Extended (150), Main (100), and Diamond (50) sets. On 2026-07-07, version 1.1 refined fair-internet-use enforcement, audited more than 1,000 blocker criteria, demoted 75 overly strict blockers, and deprecated Diamond because it was no longer the hardest consistent subset and its low solve rates made results noisy.

Further reading