Epoch tests six AI agents on its own real work and finds judgment gaps

Epoch AI graded six models on 11 of its own work tasks; frontier models handled well-defined steps but missed house standards and research judgment.

Epoch AI has published Epoch Automation Reports (October 8, 2026). It took 11 tasks from its own work, including graphics, short data articles, interactive data explorers, data-center research and research design, and ran six models through them in their native agent harnesses at the highest reasoning settings. Each output was graded by hand against employee standards. The useful finding for anyone shipping with agents is where the models broke. Claude Fable 5.1 and GPT-6 Astra handled the well-defined parts reliably, such as coding and computational analysis. They failed on the open-ended parts, which is where your own review step still has to sit.

According to Epoch, models missed implicit conventions like visual style and audience fit "even with ample reference material," including Figma files holding months of design work. Asked for a simple diagram, Fable 5.1 produced something far more information-heavy than the single chart an Epoch designer made. On research tasks, models proposed plausible directions but presented results from flawed setups as key findings. AlphaSignal's write-up highlights one case: GPT-6 Astra's 4,096-token cap truncated 61 of 280 responses, and the model then reported that artifact as a finding. Epoch says open-weight models also slipped on basic steps. Kimi K3 built a whole article on a data filter it never checked.

There is also a positive result. Epoch dropped one chart-restyling task because Fable 5.1 matched house quality on it, and the team now largely automates that step.

Treat the results with some caution. Each output had a single grader, the findings are mostly qualitative, and AlphaSignal notes that the results compare whole agent setups rather than models alone. No outside replication has turned up.

Sources