研究：花哨的 harness 让编码智能体更贵但没更好

Earlier comparisons were measuring noise

Previous studies report gaps of more than 20 points between harnesses, but those numbers almost always come from a single run, according to the authors. Pure reruns of the same setup already vary by up to six points on SWE-bench Verified. Zhipu AI, for example, advertises its own harness with a 2.4-point lead over Claude Code, a margin that falls well within that noise range. Harnesses are also often wired into the benchmark differently, making comparisons even less reliable.

Liu and Han ran five models through three harnesses in their shipped versions: Claude Code, OpenCode, and mini-SWE-agent. mini-SWE-agent gives the model only a Bash tool and skips context management. Claude Code and OpenCode offer specialized file and search tools, long system prompts, and automatic history compression.

The models were Qwen3.6, Qwen3.8, Deepseek-V4-Flash, GLM-5.3-Flash, and HY4-Preview. Claude Opus 5 ran as a reference only in Claude Code. Tests used the 45 hardest tasks from SWE-bench Verified plus a pool of 447 more. Several setups ran twice to measure noise.

All runs happened offline. In early tests with web access, 71 percent of Deepseek and Qwen model attempts pulled the correct fix straight from GitHub instead of solving the task themselves. Claude models almost never did this. The researchers discarded those results and don't claim any maker trained on the test data, but the pattern was strong enough that they cut off web access for all remaining runs.

Swapping harnesses moves results about as much as a rerun

On the larger pool of 447 tasks, Claude Code and mini-SWE-agent, the heaviest and lightest harnesses, sit less than two points apart for both tested models. Within a five-point band, they're statistically equivalent.

Two-part chart. Left: scatter plot of input tokens per task vs. pass rate on 45 hard SWE-bench tasks for six models across Claude Code, mini-SWE-agent, and OpenCode. Right: pairwise pass-rate differences between harnesses with 95 percent confidence intervals. — The three harnesses consume two to three times as many tokens as the leanest option per model, but pass rates stay within measurement uncertainty. Only OpenCode's deficit on the large task pool is statistically significant. | Image: Liu & Han

The researchers expected the biggest harness effect on the 45 hardest tasks. Instead, harnesses differ by two to five solved tasks per model, and a simple rerun of the same setup shifts the count by up to three. Sometimes mini-SWE-agent leads, sometimes Claude Code does.

The flip rates tell the same story. Between two reruns, the median outcome changes on 13 percent of tasks. Between two different harnesses, it's also 13 percent. Swapping models pushes that to 22 percent. Which tasks a given harness wins changes from run to run, matching a coin flip. The harness gap behaves like a "fresh draw of the same noise," the authors write.

Dot plot comparing the share of tasks whose outcome flips across three conditions: same-harness rerun, harness swap, and model swap. Medians land at 13, 13, and 22 percent. — Swapping harnesses flips task-level results about as often as a simple rerun, while swapping models causes much larger shifts. | Image: Liu & Han

These small samples also limit what any comparison can detect. With 45 tasks, even a 13-point gap shows up only half the time. With 447 tasks, gaps of about five points become reliably detectable. Zhipu's claimed 2.4-point advantage falls below both thresholds, and so do many other advertised improvements.

All harnesses perform the same, except OpenCode falls behind

The only measurable harness effect is a steady deficit for OpenCode. On the large task pool, it trails the other two by up to nine points. About half of that gap traces back to an output limit. When a model exceeds this limit mid-step during reasoning, Claude Code prompts it to keep going. OpenCode lets the step end there, and the attempt usually fails.

Adding a Claude Code-style continuation prompt to OpenCode didn't help, because the same task rarely triggers the same overflow on a retry. For the rest of the gap, the authors have no explanation, only an observation: under OpenCode, models stop working much sooner. On locally run models, OpenCode also had a much tighter time limit than Claude Code, which the authors flag as an uncontrolled variable.

mini-SWE-agent runs into the context window because it lacks context management, mostly on tasks where the other harnesses also fail. OpenCode's search tools try to download a binary file on first use and break when offline.

Stacked bar chart showing for each model and harness the share of passed tasks, regular failures, output-token cutoffs, context overflows, step/time limits, and infrastructure failures, split by hard45 and pool447. — With Qwen3.8, some OpenCode runs hit the output limit, while mini-SWE-agent without context management more often runs into the context window. | Image: Liu & Han

System prompt and tool descriptions set the bill

With the same model, same tasks, and same price list, a single task can cost up to three times more depending on the harness. Tool outputs barely explain the gap. All three harnesses return similar amounts of text per call.

The real difference comes from the system prompt and tool descriptions. They get resent with every step and run about 16,600 tokens for Claude Code, roughly 7,000 for OpenCode, and under 1,000 for mini-SWE-agent. Claude Code also keeps the model's reasoning output in the conversation history, based on the researchers' reverse engineering.

Two line charts showing input tokens per model call across agent steps for GLM-5.3-Flash and DeepSeek-V4-Flash, with median lines for Claude Code, mini-SWE-agent, and OpenCode. — Claude Code starts much higher because of its roughly 16,600-token system prompt with tool definitions. After that, context grows at similar rates across all three harnesses. | Image: Liu & Han

After that first call, context grows at similar rates across all harnesses. The system prompt size sets the baseline cost, and the number of steps multiplies it. Which factor dominates depends on the model: With GLM-5.3-Flash, Claude Code costs nearly twice as much as mini-SWE-agent. With Deepseek-V4-Flash, mini-SWE-agent needs so many extra steps that the two end up even. Lower prices for cached tokens dampen the gaps but don't change the ranking.

Four bar charts showing tokens and costs per trial in yuan for Claude Code, mini-SWE-agent, and OpenCode on DeepSeek-V4-Flash and GLM-5.3-Flash. — With GLM-5.3-Flash, a single trial in Claude Code costs 1.81 yuan versus 0.56 yuan in OpenCode. With DeepSeek-V4-Flash, Claude Code and mini-SWE-agent end up at roughly the same price. | Image: Liu & Han

No home-field advantage, open questions around thinking

Deepseek's own harness scores no better with its own model than the other harnesses do, but costs the most. Claude Opus 5 delivered identical results even after the researchers removed tools from Claude Code. A year-old version solves a similar number of tasks at less than half the cost, because Extended Thinking wasn't on by default back then.

Thinking was the only case where the harness choice actually changed the outcome. Without thinking, mini-SWE-agent with Qwen3.6 drops eleven points, while Claude Code and OpenCode stay within noise. HY4-Preview also loses eleven points in Claude Code without thinking. How much an agent depends on reasoning seems tied to the specific model-harness combination. The authors admit their test grid is too small to explain why.

The study relies mainly on one benchmark. A small cross-check on Terminal-Bench 2.1 shows the same pattern. The closed-source top model, Opus 5, ran only in Claude Code. A harness buys " fewer ways to lose a task, and a bill, not pass rate," the authors conclude. They recommend publishing a rerun of the same setup alongside every harness comparison. That's the only way to tell how much of a measured difference is just noise. Data and analysis scripts are available on GitHub.

More effort in the harness rarely pays off in other studies, either. In one, a simple Markdown file beat a complex skill system for AI coding agents. An earlier study already showed that the choice of framework can nearly triple costs for the same model. Harness design gets a lot of attention as a result. Nvidia recently cut nearly half its token costs for coding agents through better agent steering.