Introducing FrontierCode 2.0: our new coding benchmark that...
Cognition@cognition
11 views
Oct 28, 2026
~2 min read
Advertisement
1
Introducing FrontierCode 2.0: our new coding benchmark that measures a model’s ability to go beyond the spec.
Most coding evals measure a model’s ability to write code that meets a spec. With FrontierCode 1.0, we asked a new question: would you actually merge the PR?
FrontierCode 2.0 is comprised of more diverse tasks, featuring larger scopes and higher mergeability standards. We are now asking: not only would you merge this PR, but should you?
One key feature of FrontierCode 2.0 is underspecified prompts. Rather than including every engineering decision, prompts are deliberately ambiguous, giving the models room to demonstrate their judgment.
The result is a benchmark that is far from saturated, with the top model scoring just 45%.
Most coding evals measure a model’s ability to write code that meets a spec. With FrontierCode 1.0, we asked a new question: would you actually merge the PR?
FrontierCode 2.0 is comprised of more diverse tasks, featuring larger scopes and higher mergeability standards. We are now asking: not only would you merge this PR, but should you?
One key feature of FrontierCode 2.0 is underspecified prompts. Rather than including every engineering decision, prompts are deliberately ambiguous, giving the models room to demonstrate their judgment.
The result is a benchmark that is far from saturated, with the top model scoring just 45%.
2
On our price vs performance evaluation, models cluster into several groups.
▪️ Opus 5.5 and GPT-6 Astra lead in intelligence, at average costs of $12.8 and $6.67, respectively.
▪️ GPT-6.1 Sol leads in cost efficiency, scoring 39% at $1.03 per task.
▪️ GPT-6 Luna and Haiku 5.5 lead on overall cost, scoring above 20% at less than $0.50 per task.
Chinese OSS models lag far behind, with Kimi K3 and GLM 5.3 scoring at 17%. American OSS models like Beam and Inkling are even further behind with a score of 3%.
▪️ Opus 5.5 and GPT-6 Astra lead in intelligence, at average costs of $12.8 and $6.67, respectively.
▪️ GPT-6.1 Sol leads in cost efficiency, scoring 39% at $1.03 per task.
▪️ GPT-6 Luna and Haiku 5.5 lead on overall cost, scoring above 20% at less than $0.50 per task.
Chinese OSS models lag far behind, with Kimi K3 and GLM 5.3 scoring at 17%. American OSS models like Beam and Inkling are even further behind with a score of 3%.
3
FrontierCode 2.0 separates model tiers more clearly than FrontierCode 1.1. It also ranks them differently: on DeepSWE v1.1, Muse Wonder 1.0 and Gemini 4 Argon score highest, but on FrontierCode 2.0 they land in the middle of the pack.
4
One example task comes from Mock Service Worker, a JavaScript library for mocking network requests. The task asks the model to fix a user-reported bug: resetHandlers() doesn’t restart the generator.
Claude Opus 5.5 fixes the bug by discarding the old generator reference upon reset, so the next request starts a new one. That solves the reported bug, but dropping the reference to the generator means that any cleanup code the user already had in a finally block will never run.
The prompt doesn’t mention cleanup, but an experienced JavaScript developer knows to expect finally to run when a generator is abandoned, and Opus 5.5’s solution fails to satisfy that implicit correctness expectation.
Claude Opus 5.5 fixes the bug by discarding the old generator reference upon reset, so the next request starts a new one. That solves the reported bug, but dropping the reference to the generator means that any cleanup code the user already had in a finally block will never run.
The prompt doesn’t mention cleanup, but an experienced JavaScript developer knows to expect finally to run when a generator is abandoned, and Opus 5.5’s solution fails to satisfy that implicit correctness expectation.
5
Each task on FrontierCode 2.0 was built by a maintainer of the open-source repo it comes from, with 40+ hours of work per task. In 80% of tasks, our audits led to a reference solution that improves on the original upstream PR. One audit agent also found a security vulnerability upstream, which we reported to the maintainers.
Thank you to the maintainers who built these tasks: @streamer45 @cristiam86 @martinmck_ @Murderlon @smilingnosrati @deluccadev @iifawzi_ @patricio0312rev @kaitran_ntt @manuelkiessling @MatiPl01 and @thorbjorn81.
Thank you to the maintainers who built these tasks: @streamer45 @cristiam86 @martinmck_ @Murderlon @smilingnosrati @deluccadev @iifawzi_ @patricio0312rev @kaitran_ntt @manuelkiessling @MatiPl01 and @thorbjorn81.
6
7
Read about the methodology and scoring criteria on our blog, where we also share sample tasks:
cognition.com/blog/frontier-…
cognition.com/blog/frontier-…
