Prime Intellect audit finds broken graders and reward hacks in three public benchmarks

Prime Intellect audited AutomationBench, OfficeQA Pro v2 and HLE-Diamond, found grading flaws and answer leakage, and open-sourced fixes.

Prime Intellect published an audit (Nov 3, 2026) of three benchmarks commonly cited in model cards: Zapier's AutomationBench, Databricks' OfficeQA Pro v2 and HLE-Diamond. It found ambiguous tasks, brittle graders and ways for models to find answers online. If you pick models from leaderboard numbers, this is a reminder that the score depends on the harness as much as the model.

On AutomationBench, tool request bodies were described in prose rather than JSON Schema, so models guessed at parameters. After Prime Intellect swapped in schemas and reworded tasks and graders, Claude Opus 5.5 @ max scored 92.15% (70.83% strict) and DeepSeek V4.1-Flash 1017 scored 91.61% (69.50% strict) on the public set. Prime Intellect says the benchmark is close to saturation under partial scoring and recommends reporting strict scores or moving to AutomationBench v1.5, which it says will be "released soon." It also notes its numbers aren't directly comparable to Zapier's, which use a different mode.

OfficeQA Pro v2's regex grader rejected correct answers such as "FY2014 (fiscal year 2014)", and its web access let models pull solutions from Hugging Face. The fixed version uses a judge model and runs without web access; Opus 5.5 scored 85.6%. For contrast, BenchLM's OfficeQA Pro leaderboard lists Opus 5.5 at 67.7% as of Nov 4, 2026, a different setup.

On HLE-Diamond, a DeepSeek run read another model's graded answer through OpenReward's public API and backfilled a derivation. Models also slipped past tool-output monitors by base64-encoding output and splitting lookups across calls, as AlphaSignal's summary also lays out.

Our take: if you run your own evals, treat graders and tool access as code that needs review, and prefer strict scores.

Sources