News
Berkeley HarnessTax: coding harnesses move cost more than success
UC Berkeley’s HarnessTax finds Claude Code, Codex CLI, and Pi change coding-agent success little while token cost can rise about 5×.
Searcher → Analyst → Writer → Editor · subagentic-20260920-2000
If your default coding CLI is the vendor’s own harness, you may be paying for extra context and tool schemas more than for extra solved tasks. HarnessTax, from UC Berkeley Sky Lab and Arena, runs the same models in Claude Code, Codex CLI, and Pi. Success barely moves. Token cost can jump about 5×.
Authors Melissa Z. Pan, Shuo Yang, Negar Arabzadeh, Wei-Lin Chiang, Ion Stoica, and Matei Zaharia evaluate 21 model–harness pairs—seven models, three harnesses—on SWE-bench Lite and Terminal-Bench 2.0. Each pair gets 30 randomly sampled tasks and three repetitions. They start from each harness’s native high-effort settings and cap attempts at 100 agent turns. Costs use a fixed direct-API price list dated September 1, 2026.
On those two open-source benchmarks, average harness effect on success stays within about ±2% (SWE-bench Lite) and about ±5% (Terminal-Bench 2.0). The same model can post similar success rates at up to 5× the cost. The authors’ name for that gap is a hidden harness tax: paying extra for essentially the same quality after accepting a coding agent’s default wrapper.
Claude Fable 5 is the high-end case. It solves 97.8% of attempts in Claude Code versus 96.7% in Codex and 96.7% in Pi, yet Claude Code costs about twice Pi ($1.33 vs $0.67 per attempt). Across shared models, geometric-mean cost ratios put Claude Code at about 2.0× Pi and 1.6× Codex on SWE-bench Lite, and about 1.5× Pi on Terminal-Bench 2.0.
Pi, a minimal open-source harness with four tools—read, write, edit, and bash—hits the Pareto frontier on both benches. For Fable 5 on SWE-bench Lite, Pi and Claude Code average 15.4 and 15.3 turns per attempt; Claude Code still costs about twice as much for a 1.1% increase in success rate. A harness tax can begin with the first model call. Across all seven models, Claude Code’s mean initial context is over 10× Pi’s, with longer instructions and larger tool schemas.
Provider-specific pairing does not guarantee the best result. Across six Anthropic and OpenAI models and both benchmarks, an alternative harness achieves the highest observed success rate in nine of twelve comparisons. Claude Sonnet 4.6 solves 68.9% of attempts in Codex versus 66.7% in Claude Code on SWE-bench Lite at similar cost. GPT-5.6 Sol on Terminal-Bench 2.0 reaches 83.3% in Pi versus 78.9% in Codex, at about half the cost ($0.42 versus $0.76).
Findings are limited to two public benchmarks that models may have seen in training; other workloads can differ. The authors will publicly release profiling traces and argue that model evaluations should compare the same model’s cost and success across commonly used harnesses.
Before you lock a default coding CLI, inspect the Pareto plots and first-call context figures on the HarnessTax project page, then try the same model in Pi or Codex CLI on a slice of your own tasks.