We ran 5 coding-agent harnesses on one model, DeepSeek V4.1 Flash,...
Merge@merge_api
36 views
Sep 30, 2026
~2 min read
Advertisement
1
We ran 5 coding-agent harnesses on one model, DeepSeek V4.1 Flash, across the 30 SWE-bench Lite tasks.
The harness alone moved success from 61% to 75% and cost per attempt by 3.2×.
Two agents even tried to cheat their way out of the sandbox, see how 🧵
The harness alone moved success from 61% to 75% and cost per attempt by 3.2×.
Two agents even tried to cheat their way out of the sandbox, see how 🧵
2
The harness changed how often the same model solved a task.
On the 28 tasks all five ran, DeepSeek Harness solved 75%, Claude Code 68%, Grok Build 68%, Codex 64%, Pi 61%.
Only DeepSeek Harness's lead over Pi is clearly more than chance.
On the 28 tasks all five ran, DeepSeek Harness solved 75%, Claude Code 68%, Grok Build 68%, Codex 64%, Pi 61%.
Only DeepSeek Harness's lead over Pi is clearly more than chance.
3
Those higher scores cost more.
Per attempt, DeepSeek Harness $0.244, Claude Code $0.235, Grok Build $0.117, Codex $0.099, Pi $0.075.
The two most accurate harnesses cost 3.2× and 3.1× what Pi did, and Claude Code cost nearly as much as DeepSeek Harness with fewer solves.
Per attempt, DeepSeek Harness $0.244, Claude Code $0.235, Grok Build $0.117, Codex $0.099, Pi $0.075.
The two most accurate harnesses cost 3.2× and 3.1× what Pi did, and Claude Code cost nearly as much as DeepSeek Harness with fewer solves.
4
The gap narrows to 2.5× once you count cost per fix.
DeepSeek Harness $0.33, Claude Code $0.31, Grok Build $0.16, Codex $0.15, Pi $0.13.
Solving more tasks spread the extra spend, but the two priciest harnesses still pay about twice what the other three do for each fix.
DeepSeek Harness $0.33, Claude Code $0.31, Grok Build $0.16, Codex $0.15, Pi $0.13.
Solving more tasks spread the extra spend, but the two priciest harnesses still pay about twice what the other three do for each fix.
5
The same model also ran at very different speeds.
Codex 239s, Claude Code 422s, Pi 481s, DeepSeek Harness 533s, Grok Build 587s.
Codex's median solve took about 4 minutes and Grok Build's nearly 10. Times come from emulated x86 containers.
Codex 239s, Claude Code 422s, Pi 481s, DeepSeek Harness 533s, Grok Build 587s.
Codex's median solve took about 4 minutes and Grok Build's nearly 10. Times come from emulated x86 containers.
6
With the internet blocked, agents went looking for the answer anyway.
Pi and Codex searched git history for the fix. Codex and DeepSeek Harness found the model proxy, beat its vendor pin and got 21 answers from other models.
Neither passed and both were excluded from the results.
Pi and Codex searched git history for the fix. Codex and DeepSeek Harness found the model proxy, beat its vendor pin and got 21 answers from other models.
Neither passed and both were excluded from the results.
7
Picking a harness is as big a choice as picking a model.
For @deepseek_ai's V4.1 Flash, swapping the harness moved success 14 points, cost 3.2×, and speed 2.5×.
Test your favorite harness on Merge Gateway today: gateway.merge.dev
Thanks @arena for the inspiration!
For @deepseek_ai's V4.1 Flash, swapping the harness moved success 14 points, cost 3.2×, and speed 2.5×.
Test your favorite harness on Merge Gateway today: gateway.merge.dev
Thanks @arena for the inspiration!
