🔍 Read the full analysis: FrontierCode: What A Maintainer Needs To See In An AI Pull Request on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis evaluated 23 models on FrontierCode, a coding benchmark that scores whether maintainers would accept a patch, not just whether tests pass. Claude Opus 5.5 led with a 52.2% score and passed all blocking checks in 57.7% of attempts; results show substantial differences in cost and performance.
Artificial Analysis tested 23 AI models on FrontierCode, a benchmark that evaluates whether a coding agent’s patch would meet repository maintainers’ standards for acceptance. Claude Opus 5.5 ranked first with a 52.2% score, but passed every blocking check in only 57.7% of attempts, indicating that human review remains necessary under this evaluation.
FrontierCode’s results were added to Artificial Analysis’s evaluations on October 12, 2026. Its 100-task Main set draws on tasks from repositories including Celery, Budibase, uppy and Mattermost. The benchmark’s full collection contains 150 tasks. According to the source report, the tasks are private, and more than 20 maintainers contributed task design.
The leading models scored closely: GPT-6.1 Sol placed second at 49.1%, followed by Claude Fable 5.1 at 49.0%. Claude Sonnet 5.5 scored 45.8%, while the open-weights MiMo-V2.6-Pro reached 45.4%. Artificial Analysis used the same open-source agent harness, Stirrup, for each model, with up to 250 turns per attempt and three attempts per task.
FrontierCode checks patches on multiple dimensions, including hidden tests, regression and cleanliness commands, test quality, scope limits and an LLM review against maintainer criteria. Failing any blocking check gives an attempt a zero; passing all blocking checks earns a score from 70% to 100%, depending on quality checks. The source report says nine of the 100 tasks were unsolved by every model on every attempt.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Tests Is Not Enough
Many coding evaluations focus on whether generated code passes tests. FrontierCode asks a broader question: whether the change is correct, appropriately scoped and useful to maintainers. That distinction matters because a patch can pass visible tests while adding weak tests, changing unrelated files or creating work for reviewers.
The results suggest that benchmark performance should not be read as proof that an agent can safely handle pull requests without oversight. Even the top model failed at least one blocking check in a substantial share of attempts. That does not establish how models perform across all real-world repositories, but it does show that success on this benchmark is not yet consistent.
The report also compares the results with Artificial Analysis’s Intelligence Index and finds that model rankings can shift when judged on code review criteria. For example, Claude Sonnet 5.5 ranks fifth on FrontierCode, while Grok 4.7 ranks 18th. A general model leaderboard may not predict which system produces the most maintainable repository changes.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Scores Patches
FrontierCode was created by Cognition and open-source maintainers and announced in June 2026. Version 1.1 followed in July with rules against searching online for task solutions, according to the source report. Maintainers spent more than 40 hours on each task, which was drawn from a real repository.
Its checks include reverse tests, which assess whether an agent’s own tests would fail against the original code, and adaptive tests for complex behavior. The benchmark also penalizes overly large diffs and edits outside relevant files. Cognition’s trajectory scanner can flag attempts to look up solutions; the report says those attempts receive a zero.
Artificial Analysis’s reported costs varied widely. GPT-6 Luna (max) scored 41.9% at $0.09 per task, while MiMo-V2.6-Pro scored 45.4% at $0.18 and GPT-6.1 Sol scored 49.1% at $0.81. Claude Opus 5.5 scored 52.2% at $10.15. These figures are benchmark costs, not a full accounting of deployment or review expenses.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, in Cognition’s announcement, as quoted by ThorstenMeyerAI.com
As an affiliate, we earn on qualifying purchases.
Limits of the Current Results
The scores apply to Artificial Analysis’s 100-task Main set and its specified harness and attempt limits. They do not establish how models will perform across every codebase, task type or development workflow. The report does not provide enough information here to treat the measured task costs as typical costs for production use.
Several models were missing from the leaderboard as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their eventual results could change the ranking. The supplied report also gives no independent validation of the benchmark’s scoring judgments or detailed breakdown of individual model failure types.
As an affiliate, we earn on qualifying purchases.
More Model Results to Watch
The next visible development is whether the missing models receive FrontierCode results and how their scores compare on the same Main set. Further reporting could also clarify performance by repository and task category, alongside how often models fail particular blocking checks.
For now, the results provide a comparative benchmark rather than a deployment guarantee. Teams considering AI-generated pull requests can use the findings to distinguish test completion from review readiness, while keeping human review in place for changes generated by these systems.
software testing and validation tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
It evaluates AI-generated code changes against tests, regression and cleanliness checks, test quality, file and diff scope, and an LLM review based on maintainer criteria.
Which model ranked first?
Claude Opus 5.5 (max with fallback) ranked first with a 52.2% FrontierCode score. It passed all blocking checks in 57.7% of attempts.
Does a high score mean an AI can merge code without review?
No. The reported scores measure results on a defined benchmark, and even the top model did not pass every blocking check in 42.3% of attempts. The results do not establish that unreviewed AI pull requests are safe.
How much did the tested models cost per task?
Reported costs ranged from $0.09 per task for GPT-6 Luna (max) to $10.15 for Claude Opus 5.5. These are benchmark task costs and may not represent total costs in a development workflow.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
