A coding agent’s harness can triple its cost without making it more accurate, study finds

VentureBeat · Ben Dickson · 2026-10-23 21:53 UTC

Researchers held five models fixed across three production harnesses. A vendor’s own harness bought no edge, and the only effect that beat the noise was a harness losing tasks.

The harness surrounding an AI coding model — its system prompt, tools and context-management rules — can triple the cost of solving coding problems without producing a measurable improvement in accuracy, according to new research. AI vendors regularly advertise accuracy gains from harness updates. But whether a more elaborate harness actually makes an agent better at solving coding problems is less clear.

Researchers at Shandong University put that question to the test. They compared three production harnesses by holding the underlying models fixed and testing them on nearly 500 software engineering tasks. On the larger task set, the most elaborate and most minimalist harnesses produced statistically equivalent success rates within a five-percentage-point margin, although a third harness performed worse.

On the 45 hardest tasks, switching harnesses changed which problems the agents solved about as often as rerunning an unchanged setup: 13% of tasks in either case, measured at the median. Meanwhile, the researchers found that cost per task varied by as much as threefold across harnesses using the same model.

Read the full story at VentureBeat · All coverage of this story