The harness is the product. What the agent leaderboards cannot tell you about it
The scaffolding around a model now has a name. It explains why the same model feels brilliant in one tool and useless in another, and why the board everyone quotes cannot separate the two.
A language model is a function from text to text. Everything that decides which text it sees, which tools it may call, what happens to each result and when the job is finished sits outside the model, and the word for that layer has settled into harness. Anthropic describes its Agent SDK as giving you the same tools, agent loop and context management that power Claude Code.
This matters commercially because the same weights ship inside many different products. When a model feels sharp in one tool and hopeless in another, the harness is usually the difference: what it loads into the context window, what it discards, whether it retries, whether it checks its own work, and how much it is allowed to do before it asks you.
The tempting next claim is that the harness now matters more than the model. The public evidence does not support the slogan, and the leaderboard people quote most cannot settle it, because almost every entry pairs a lab's own harness with that lab's own model. What the board does show is more useful than the slogan anyway.
What a harness is, and what the Terminal-Bench 4.0 board actually shows
1. A harness is six jobs and a loop
The clearest definition comes from a benchmark paper rather than a vendor. Harness-Bench calls it the system layer managing context, tools, state, constraints, permissions and recovery. Around those sits the loop: the model emits a tool call, the harness executes it, feeds the result back, and decides whether to continue or stop. That loop is what turns a text predictor into an agent, and it is code somebody wrote, not a property of the model.
1. A harness is six jobs and a loop: verified pricing, fit and cautions →
2. Context management is where most harnesses win or lose
A long task overruns any context window, so every harness needs a policy for what to forget. The two mechanisms are not the same: compaction summarises earlier context into a shorter form, while context editing clears old tool results or reasoning outright. Summarising keeps a lossy trace of everything; clearing keeps the recent turns exact and loses the rest completely. Which one a product chose is rarely documented, and it decides what the agent can still see two hundred turns into a job.
3. The leaderboard ranks bundles, not models
On the Terminal-Bench 4.0 board as we read it on 7 September 2026 there are eighteen entries, each run over 330 trials. Fifteen of them pair a lab's own harness with that lab's own model: Claude Code with Anthropic models, Codex with OpenAI models, Grok Build with Grok. Only three rows cross the line, and two of those are the deliberately minimal mini-SWE-agent running Gemini Flash models. A board built this way can rank products. It cannot tell you how much of a score belongs to the harness.
4. The top of the board is a statistical tie
Codex on GPT-6 Astra at max effort leads with 58.18%, and the published 95% interval is plus or minus 2.79 points. Claude Code on Fable 5.1 scores 57.88% with an interval of plus or minus 3.76. Codex at xhigh and at high both also score 57.88%. Four entries, one margin of error, no winner. Anyone quoting the rank order of the top four as a capability ranking is reading noise as signal.
5. Effort is a dial with a bill attached, and it is not the harness
The only controlled comparison on the whole board is one model in one harness at five reasoning-effort settings. Codex on GPT-6 Astra scores 58.18% at max for $3,267 of compute, and 57.88% at high for $2,269. That is 44% more money for three tenths of a point, comfortably inside the error bar. Medium gives 54.24% at $1,915 and low gives 50.61% at $1,557. Before anyone rebuilds a harness, that single configuration field is the cheapest lever in the room.
6. A bad pairing costs more, not less
The intuition that a weaker setup is at least cheaper is wrong here. Claude Code on Sonnet 5 scored 12.42% while spending $9,604 and 21.6 billion tokens, averaging 6,510 seconds per trial. Codex on GPT-6 Astra at max scored 58.18% on $3,267 and 1.5 billion tokens at 2,796 seconds. Roughly a third of the money, a fifth of the tokens, and nearly five times the score. When an agent cannot solve something, the loop does not stop; it keeps buying tokens until a limit does.
7. The gap between first-try and fifth-try is the harness's playing field
Terminal-Bench publishes pass rates at several attempts. Claude Code on Fable 5.1 solves 57.9% of tasks first try and 78.8% within five; Codex on GPT-6 Astra at max goes from 58.2% to 71.2%. That is roughly twenty-one points of headroom in one case and thirteen in the other, sitting there for any harness that can run several attempts and reliably tell which one worked. Closing that gap is verification and orchestration, not intelligence, and it is the most concrete thing a harness can actually buy you.
8. Harnesses port across vendors, and that is the interesting part
One row on the board is Claude Code driving Z.ai's GLM-5.3 to 41.82%, at $2,728 against $5,969 for the same harness on Anthropic's own Opus 5, which scored 51.82%. Cheaper and lower, on the same scaffolding. Open weights make the model a component you can swap under a harness you did not build, which is the same portability that lets a European provider serve a Chinese model on its own hardware.
9. The research and the vendors agree: report the pair
Harness-Bench ran 106 sandboxed tasks and 5,194 trajectories and concluded that agent capability should be reported at the model-harness configuration level rather than attributed to the base model alone. It also named the failure mode it saw most, execution-alignment failures, where the model's reasoning drifts away from what the tools and the workspace are actually telling it. Anthropic says something adjacent from the other side: that its default harness is built for coding and that research, security analysis and code review needed custom ones.
So the honest version is not that the harness beats the model. It is that capability is a property of the pair, that almost nobody publishes numbers which let you separate them, and that the two cheapest improvements available right now are not architectural at all. Turn the effort dial to the level where the score stops moving, and give the agent a way to check its own work so some of that pass-at-five headroom lands on the first attempt.
If you are choosing between agent products, this changes the question you ask. Not which model does it use, but which model at which effort, inside which loop, with what context policy, what tools, and what it may do without asking. Then run your own tasks and record cost and tokens next to the pass rate, because a setup that fails expensively is the specific failure these numbers keep showing.
Questions people ask
- What is an AI harness?
- The harness is the software wrapped around a language model that makes it an agent: the loop that executes tool calls and feeds results back, the tool set, context management, permissions, state and error recovery. The model supplies judgement; the harness decides what it sees, what it can do and when it stops.
- Is the harness the same thing as the model?
- No, and the distinction is practical. The same model weights ship inside many products with different harnesses, which is why one model can feel capable in one tool and hopeless in another. It also cuts the other way: swapping a stronger model into a weak pairing does not automatically fix it.
- Does the harness matter more than the model?
- The evidence does not settle it. Public agent leaderboards mostly pair each lab's harness with its own models, so the two cannot be separated. What the current Terminal-Bench 4.0 board does show is that reasoning effort, a single configuration setting, moves scores by several points and costs by tens of percent, and that a poor model-harness pairing burns far more money than a good one.
- What is the difference between the Claude Agent SDK and the Claude API?
- The API is the model endpoint, where you write the tool loop yourself. The Agent SDK is the Claude Code harness packaged as a library, supplying the loop, built-in file and command tools, context management, permissions and subagents. Both run on infrastructure you host; Anthropic's Managed Agents is the option where Anthropic runs the loop and the sandbox too.
- How do I compare two AI coding agents fairly?
- Hold everything constant except the one variable you are testing, because effort setting, model version and date all move the numbers as much as the harness does. Use tasks from your own repository, run each several times, and record cost, tokens and wall-clock time beside the pass rate. A public leaderboard tells you about a bundle somebody else assembled, not about yours.
All leaderboard figures were read on 7 September 2026 directly from the Terminal-Bench 4.0 leaderboard at tbench.ai, including accuracy, published 95% confidence half-widths, pass-at-k rates, total cost in US dollars, token totals and average trial duration; each entry is over 330 trials. Important caveat stated in the piece and repeated here: those eighteen rows are not a controlled experiment. Submission dates run from May to September 2026, reasoning-effort settings differ between rows, and most rows pair a lab's own harness with its own model, so only the five-row Codex and GPT-6 Astra effort sweep isolates a single variable. Definitions and findings attributed to Harness-Bench come from the paper "Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows" (arXiv, submitted 27 May 2026), which is a preprint we have not seen peer reviewed. Anthropic's descriptions of the Agent SDK come from its own Agent SDK overview documentation, and the points about custom harnesses and subagent failure modes from its dynamic workflows post of 2 June 2026; both are vendor material about vendor products. We have not reproduced any benchmark result ourselves. Correction to a common framing this article started from: the claim that the harness now matters more than the model is not supported by the public evidence, because the leaderboards that would demonstrate it mostly cannot separate the two. Disclosure: TaskNorth's library holds records for Claude Code, Claude, ChatGPT, Cursor, GitHub Copilot, Grok and others named here, and our recommendations draw on that library. We have no disclosed commercial relationship with Anthropic, OpenAI, Google, xAI or Z.ai.
Trying to work out which AI tools fit your task? Describe the outcome and get a Blueprint: the tools, the prompt, and the steps, with pricing we verified ourselves.