Intelligent Networks Boost Distributed AI Training Across WANs

Distributed training of AI models across geographically dispersed data centers, connected by wide area networks (WANs), faces significant hurdles. The continuous exchange of parameters between compute clusters is often bottlenecked by limited bandwidth, high latency, and uneven network topologies. This makes achieving the efficiency of co-located training challenging. Researchers are proposing a novel approach that transforms the network into an active, intelligent component of the training process. This involves two key system-side innovations: utilizing multicast technology to efficiently replicate outbound data traffic and deploying in-line FPGAs to aggregate inbound traffic. These technologies, typically used within data centers, are extended to the WAN to alleviate egress and ingress bottlenecks. Complementing these system enhancements, an optimization framework has been developed. This framework generates sophisticated synchronization schedules, such as rotating cliques of compute islands, tailored to the underlying network topology and the capabilities of these intelligent network technologies. Demonstrations on a nine-city network, modeled after a live programmable WAN, illustrate how these optimal schedules dynamically adjust to network conditions, effectively closing the performance gap with the ideal scenario of co-located training.