DeadPool Enables Resilient LLM Training with Hot-Swapping.

Training large language models (LLMs) is an incredibly resource-intensive process, often requiring tens of thousands of GPUs running for months. During such extended operations, failures across both software and hardware stacks are common. Existing fault-tolerance solutions either introduce significant performance overhead during normal, error-free execution or lead to unacceptably long recovery times, especially when a few compute nodes permanently fail. This research introduces DeadPool, a novel system designed to overcome these limitations by simultaneously optimizing for both zero overhead during failure-free operation and extremely fast recovery. DeadPool's core innovation lies in its ability to restore LLM training through "hot-swapping," which means replacing failed nodes with spare ones without halting the entire training job. This capability is powered by two key ideas. First, it employs an in-memory checkpointing mechanism that operates off the critical path, providing spatial redundancy without impacting training speed. Second, it features a communicator reconstruction protocol that dynamically replaces failed nodes with spares at runtime. The system efficiently overlaps its in-memory checkpointing with ongoing computation, ensuring no performance penalty during error-free execution. In the event of permanent node failures, DeadPool can reconstruct memory states with minimal recomputation by leveraging these in-memory checkpoints. Experimental evaluations, conducted at scales up to 512 NVIDIA A100 GPUs and with LLMs up to 65 billion parameters, demonstrated zero checkpoint overhead and hot-swapping recovery completing in under 40 seconds. These results highlight DeadPool's effectiveness in achieving both high efficiency and robust fault tolerance for large-scale LLM training.