AgentWorld benchmark: under a third of multi-agent team actions contribute to success

A new open-source benchmark finds the best tested model reaches 52% success on long-horizon multi-agent tasks, with most agent actions not advancing the goal.

A new paper, AgentWorld (arXiv, September 25, 2026), tests how well teams of 3–20 LLM agents collaborate over long tasks in an MMORPG sandbox. The best model it tested, Gemini 3 Flash, reached 52.0% task success. If you're thinking about splitting a workflow across several agents, the more useful number is a second one. By the authors' metric, less than a third of that team's actions actually contributed to finishing the task.

The benchmark has 100 human-annotated tasks plus 100 augmented variants. The tasks run 25–55 rounds, and agents can't see each other's internal state, so they have to coordinate through messages and shared plans. The paper's full text introduces Causal Collaboration Effectiveness (CCE). CCE traces backward from a successful outcome to measure which actions causally mattered. The best model scored a CCE of 0.320. The authors report recurring failures: communication breakdowns, role confusion and losing shared plans across rounds.

An independent summary on GitHub lists the other results: Claude Haiku 4.5 at 45%, GPT-5 Mini at 36% and DeepSeek R1-70B at 20%. It also notes that communication volume didn't predict success. The same summary flags three limits:

No independent replication has turned up. It's also unclear whether CCE can score agent traces from outside the sandbox.

For comparison, a DEV Community post recommends checking four things before adding agents: whether the subtasks are truly parallel, whether the cost is justified, whether every agent has a written brief, and whether there's a named verifier. Otherwise, the post argues, a single well-briefed agent usually wins.

Sources