Google DeepMind Trains 12B-Parameter Model Across Four Regions With Decoupled DiLoCo
Google DeepMind published Decoupled DiLoCo, a method for training large AI models across regions on low-bandwidth networks.
Google DeepMind published research on Decoupled DiLoCo, a distributed training method designed to run large-scale AI model training across multiple data centers without requiring high-bandwidth interconnects or uniform hardware. The team trained a 12B-parameter Gemma model across four US regions over low-bandwidth networks and demonstrated the ability to mix hardware generations — TPU6e and TPUv5p — in a single training run without performance degradation.
The system is also self-healing: during testing, artificial hardware failures were injected mid-run, and Decoupled DiLoCo isolated the disruptions, continued training, and reintegrated offline units when they came back online. The method builds on two prior Google efforts — Pathways, which links heterogeneous chips into a shared compute fabric, and DiLoCo, which minimizes inter-datacenter bandwidth requirements. For AI developers and infrastructure teams, the research points toward training architectures less dependent on co-located, homogeneous hardware clusters — a meaningful constraint in scaling frontier models today. Full details are available in the Google DeepMind thread.