FrontierHarness: same model, 12 harnesses, ~17× median cost spread

If you hold the model still, the harness still moves the score. FrontierHarness, a dashboard at frontierharness.org, compares twelve coding-agent loops on a shared Runta restore and finds pass rates from 50.0% to 66.7% and a roughly 17× spread in median cost per task — $1.05 on the cheap end, $18.34 on the expensive one.

That is a single-lab bake-off, not a peer-reviewed leaderboard. Guanlan Dai, posting as @guanlan on September 2, 2026, framed the result as a shift in the question itself: “A year ago the question was which model. Now it's which harness.” The post says the runs used the same model, the same tasks, and the same runtime — 360 runs and 2 billion tokens — with pass rates from 50% to 67%. It labeled the $1.05–$18.34 band as “cost per pass.” The dashboard itself titles those dollars median cost per task. The 17× figure is spend per task, not spend per successful pass.

Twelve loops, one restore

On the live table, Codex is the quality leader: 66.7% pass rate, $3.47 per task, 6 minutes 43 seconds median runtime (v0.148.0). Claude Code matches DSH Creator at 63.3%, but the bills do not. Claude Code (v2.1.237) sits at $18.34 per task and 9 minutes 38 seconds; DSH Creator is $3.28 and 6 minutes 44 seconds.

Pi (v0.84.2) is marked the balanced pick: 60.0% at $2.43. Two other DeepSeek Harness variants, DSH PTC and DSH Standard, also land at 60.0%, at $4.58 and $3.46.

Mid-pack, Oh My Pi (v17.4.0), Kimi Code (v0.37.2), and DSH Minimal all post 56.7%. DSH Minimal is the speed leader at 5 minutes 41 seconds, with a $4.72 task cost. Kimi Code is $3.65; Oh My Pi is $4.75.

Exo Harness (v0.1.0) is the cost leader: $1.05 per task at 53.3% pass, 6 minutes 17 seconds. OpenCode (v1.18.19) and Hermes (v0.20.4) both sit at 50.0%, at $3.24 and $2.90.

That is twelve named harnesses, not nine. Dai’s post listed “Pi, Exo, Claude Code, Codex, DeepSeek Harness and 4 others.” The dashboard splits DeepSeek Harness (v0.1.0-rc.8) into Creator, PTC, Standard, and Minimal, and names Oh My Pi, Kimi Code, OpenCode, and Hermes alongside them.

Headline cards on the site, rounded as shown:

Codex — 66.7% · $3.47 · 6m 43s
DSH Creator — 63.3% · $3.28 · 6m 44s
Claude Code — 63.3% · $18.34 · 9m 38s
Pi — 60.0% · $2.43 · 7m 33s
DSH PTC — 60.0% · $4.58 · 7m 44s
DSH Standard — 60.0% · $3.46 · 6m 17s
Oh My Pi — 56.7% · $4.75 · 6m 46s
Kimi Code — 56.7% · $3.65 · 7m 56s
DSH Minimal — 56.7% · $4.72 · 5m 41s
Exo Harness — 53.3% · $1.05 · 6m 17s
OpenCode — 50.0% · $3.24 · 6m 27s
Hermes — 50.0% · $2.90 · 6m 58s

Which model sat inside those loops is not named in the dashboard prose fetched for this piece.

Cost per task is not cost per pass

The site warns against reading the cheap column as a free lunch. One note: OpenCode’s low “failures excluded” figure covers only 15 passes; count failed attempts and the number becomes $3.24 per task. Another: cache hit rate is not cost — “A cached 300-turn failure can still burn more than a short cache miss.” A third: quality and cost can diverge. Claude Code “passes 19 tasks, but reaches $18.34 in cost per task.”

Those notes are why the median-cost-per-task axis is the one to quote. Exo’s $1.05 and Claude Code’s $18.34 are the ends of that axis (finer figures on the same ranking: $1.0452 and $18.3368). A separate ranking of much smaller dollar figures is not the 17× spread. The unit of that smaller-dollar ranking is not labeled in the fetched dashboard prose.

Golden checkpoint, SWE-shaped tasks

FrontierHarness v1.0, the site says, “focuses on software engineering contexts and terminal-based tasks. It may not generalize to other areas of knowledge work.” Evaluation is on Runta agent runtimes. All harnesses and the task environment are prepared once as a golden checkpoint. Every run is a fresh restore with identical vCPU, memory, disk size, disk contents, and memory state.

That protocol is the point of the comparison: the loop changes, the restore does not. No other lab is cited. Treat the ranking as Runta’s own harness bake-off on its own checkpoint, not as an independent replication, and not as a claim that the same spread would show up outside terminal-style software engineering.

The loop is part of the model

If harness choice can move pass rate from 50% to 67% and spend from about $1 to about $18 on identical tasks, model bake-offs that ignore the loop are incomplete. Codex’s 66.7% at $3.47 is a different operating point than Exo’s 53.3% at $1.05 or Claude Code’s 63.3% at $18.34 — same restore, different harness. Comparing “the model” without naming Codex v0.148.0 versus Claude Code v2.1.237 versus Exo v0.1.0 mixes the weights with the agent loop that spends against them.

Open the live table and read pass rate next to median cost per task, not in isolation. If you maintain a harness, the same page offers $100 in Runta credits to run it on the golden checkpoint. Pair that with Dai’s post for the 360-run, 2-billion-token claim — and keep the cost-per-task versus cost-per-pass labels straight.