HarnessRouter pitches Harness Arena for comparing agent harness and model combinations

HarnessRouter's open-source agent-harness server is being promoted for Harness Arena, which runs one task across harness × model configs and compares results.

HarnessRouter Community Edition is a self-hosted, Apache-2.0 server that runs agent harnesses such as Codex, Claude Code and Hermes behind one OpenAI Responses-compatible API. It is now being promoted for Harness Arena, which runs one task across several harness × model configurations and reports cost, time and pass/fail. If you pick coding agents by model alone, this lets you test the harness on your own workload.

According to @_avichawla on X (Nov 4, 2026), every config gets the same prompt, files, tools, skills and evaluation checks. Cost and time are compared only among configs that pass. The post says Arena is part of the HarnessRouter project. The repo page we read shows an eight-configuration comparison on one task but doesn't explain how the checks are defined.

The headline numbers are published by the vendor and come from a single task. The X post cites Hermes + GPT-5.2 passing 5 of 5 runs on 0.47 credits, against Claude Code + Opus 4.8 passing 3 of 5 on 223 credits. Daily Dose of Data Science (Oct 31, 2026) reports a 32% to 53% credit cut from switching only the harness on a healthcare prep task, and adds that "your numbers will differ." Per the README, setup needs Docker, about 4 GB of disk and your own provider API key.

There are caveats. tomrochette.com's review of the protocol behind HarnessRouter counts five spec versions in about ten weeks. It also notes that the steward sells the hosted service, and it found little independent coverage. Independent harness tests are humbling too. Efrain Garay's run compared seven harnesses on MiniMax-M3. He found no difference in pass rates that survived statistical correction, though cost varied nearly threefold. We'd treat Arena as a way to run that comparison yourself, not as a verdict.

Sources