Harness Leaderboard compares nine coding-agent harnesses on open models; a bash-only agent tops it

Joel Niklaus released the Harness Leaderboard, which compares nine coding-agent harnesses running open models. In each comparison only the harness changes: the model, provider and reasoning effort stay fixed, and every task gets three attempts. Niklaus says he ran 28,849 trials for $4,746 of inference, using GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint on Terminal-Bench 2.1 and DeepSWE. The post had 625 points on X by November 6.

According to Niklaus, averaged over the three models:

• mini-swe-agent, a bash-only agent, solves 51.4% of tasks; OpenClaw solves 27.3%. Four other harnesses are statistically tied with mini-swe-agent.
• Pi, one of the tied four, solves 49.4% at $0.29 per solved task, about 45% less than mini-swe-agent's $0.53.
• OpenCode costs $3.35 per solved task, 11.7 times as much as Pi.
• The best harness depends on the model: mini-swe-agent on GLM-5.3-Flash, Hermes Agent on DeepSeek-V4.1-Flash and Kimi Code on Flint.

He also points to Harness Hub's Harness Lens, which documents the prompts, tools and compaction of 34 coding-agent harnesses and links each claim to its source.