VeriHarness paper: agent rollout consensus can hide errors, disagreement can expose fixes

An arXiv paper introduces VeriHarness, which reuses a fixed base model as an agentic verifier that probes both disagreement and consensus across rollouts.

If your agent harness samples several rollouts and trusts whatever they agree on, a new arXiv paper argues that's a weak check. In VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks, posted October 1, the authors report that "disagreement often exposes correct alternatives, while consensus can conceal errors." The approach uses the same base model, with no retraining and no reference answers or grading rubrics at test time. That makes it a pattern builders could adapt without new infrastructure.

According to the abstract, VeriHarness turns the generator's underlying LLM into a verifier by giving it a workspace, evidence tools and reusable verification skills. Two roles do the work:

Their findings guide which artifact is selected and how it is revised. The authors say VeriHarness posted the highest selection scores among the baselines they evaluated, across five long-horizon workspace benchmarks and two models. With evidence-backed revision, they report gains over a single rollout of 6.2 points with Gemini 3.5 Flash and 6.4 points with Claude Opus 4.8. They also say verification skills can self-improve from failure feedback. The team released roughly 26,000 rollouts, which they say cost over $100,000 to produce.

Several things are still unclear. The abstract doesn't name the benchmarks or baselines, or give margins over those baselines. It doesn't say what the verifier costs in tokens or time, or whether code or the skills library is available. @omarsar0 describes it as a Google paper. That post is the only coverage we found, and we found no independent reproduction. Treat the results as the authors' claims for now.

Sources