AI News · 5 min read
Garry Tan's harness evals: tied pass rates, 2x cost gap on Opus
YC's Garry Tan ran Claude Code, Codex, OpenClaw, Hermes and QM on the same models and 20 tasks. Pass rates tied at 71-79%; cost per solved task did not.
What Garry Tan published
On November 8, 2026, Y Combinator CEO Garry Tan posted early results from an open-source project on X ↗: "Out of curiosity I am doing some open source eval work on harnesses with frontier models and here's what is coming back so far."
The project is g-harness-evals ↗, MIT-licensed and created on November 5. Its question is one every agent buyer now asks: which harness should you wrap around a frontier model? A harness here means the system prompt, tools and memory around the model. The method holds the model fixed (Claude Opus 5.5 or GPT-6.1 Sol) and the tasks fixed, then swaps only the harness. The matrix report ↗ covers 7 harness/model cells and 560 trials: 20 public tasks, 4 repetitions each.
One detail most summaries skipped: these aren't plain coding puzzles. The report describes them as "multi-user work in a fake company (Slack, mail, approvals)," the kind of work OpenClaw, Hermes and QM are sold for.
The results
Pass rates couldn't tell the harnesses apart. Every cell landed between 71.3% and 78.8%. According to the scoreboard ↗, "14 of 20 tasks scored the same in every cell." Eleven tasks passed every time everywhere, and three never passed anywhere. The run couldn't reliably detect gaps smaller than roughly 8 to 14 points. In other words, this was a saturated test, not a tie.
Cost could tell them apart. Here's cost per solved task at list price, including spend on failed trials:
| Harness | Model | Pass rate | Cost per solve |
|---|---|---|---|
| QM | Opus 5.5 | 72.5% | $0.19 |
| Claude Code | Opus 5.5 | 76.3% | $0.22 |
| Hermes Agent | Opus 5.5 | 78.8% | $0.24 |
| OpenClaw | Opus 5.5 | 76.3% | $0.44 |
| Codex | GPT-6.1 Sol | 71.3% | $0.17 |
| Hermes Agent | GPT-6.1 Sol | 75.0% | $0.19 |
| OpenClaw | GPT-6.1 Sol | 73.7% | $0.22 |
OpenClaw on Opus cost about twice as much per solve as the others and was the slowest, at 99.4 seconds median per trial against 40.1 for Claude Code. The README traces about 81% of OpenClaw's extra cost to cache writes, because it re-sends a much bigger system prompt and bigger tool results. On Sol, most of its gap came from output: 282,492 output tokens against Codex's 32,226.
There are some smaller findings that matter if you write agent instructions:
Claude Code doesn't open skills on its own. A skill that wasn't mentioned in the prompt was opened 0 times in 30 turns. Naming it in the prompt got 30 of 30. Codex used an unmentioned skill in 30 of 30 turns.
Where a rule sits matters. In Claude Code, a rule in the system prompt or startup file was followed in 30 of 30 turns, against 18 of 30 with no rule.
Cutting prompt text can backfire. On harder tasks, removing Claude Code's memory-and-skills text together with its tool-description advice took the pass rate from 60.4% to 0%.
Read the disclosures
Tan lists his conflicts in the README, and they're worth knowing. QM is a Y Combinator project. YC is an investor in Hermes. gstack and gbrain, which appear in some side findings, are Tan's own projects. The README also notes that OpenAI is now "the key benefactor of OpenClaw," whose creator works there. Every harness ran through the same proxy with the same graders, and the receipts are in the repo so anyone can regrade. The results are labelled as dev runs on public tasks, "not held-out confirmation."
What people are saying
Most replies picked up on the saturation point. "14 of the 20 tasks scored the same in every setup, so this test can't rank harnesses yet. What it does show is cost," wrote @Cozy2054934 ↗. @KrrishKumar2028 ↗ proposed a metric we'd also like to see: "cost per false 'done'. A cheap harness that confidently ships a broken fix gets expensive in review." And @masthan_8 ↗ drew the practical lesson: "Keep your own regression tests so you can tell when a model or tool quietly gets worse."
i10x ↗ put the results next to two other harness datasets from the same week. It reported that Joël Niklaus's Harness Leaderboard found a much wider 27.3% to 51.4% spread on cheaper open models. Its read was that harness choice moves cost "far more reliably than accuracy" at the frontier.
Our take
At the frontier, pick your harness on cost, speed and how it follows instructions. Not on "which one is smarter." On these tasks, the model did the thinking and the harness decided the bill.
What we'd do with this:
On Opus 5.5, Claude Code is the safe default. It's cheap, fast and Anthropic's own. Just name your skills explicitly in prompts or CLAUDE.md, because it won't discover them by itself. Our Claude Code review covers the rest.
On GPT-6.1 Sol, Codex is the cheapest and fastest option here. See Codex vs Claude Code for the head-to-head between the two first-party tools.
If you run OpenClaw on Opus, check your bill. Doubling cost for the same pass rate adds up across thousands of runs. Trimming the system prompt and tool output is where the README says the money goes.
If cost is your main constraint, start with our cheaper Claude Code alternatives. Routers like OpenRouter's Auto Code attack the same cost problem from the model side.
Treat the pass-rate numbers as a smoke test until a harder task set separates the harnesses. The cost numbers are already useful today.
Sources
Garry Tan on X ↗, November 8, 2026
garrytan/g-harness-evals on GitHub ↗: README, scoreboard ↗ and matrix report ↗, checked November 9, 2026
i10x: Coding agent harness vs. model ↗, November 9, 2026