ERASE Speeds Up Training for Recommendation Systems by Overlapping Passes.

Training large-scale recommendation systems, especially those with lightweight proxy models, often leads to underutilization of modern accelerators. Traditional training methods schedule forward and backward passes as distinct, sequential phases, which creates idle capacity. Researchers propose ERASE (EaRly bAckpropagation SchEdule), a new training primitive that reinterprets the detachment mechanism. By detaching a block's output, its downstream gradient dependencies are removed, allowing its backward pass to begin as soon as its forward pass completes. ERASE then launches this detached subgraph's backward pass early on a separate CUDA stream, enabling it to overlap with the forward work of subsequent blocks. This overlapping strategy was demonstrated on a lightweight transformer and, more significantly, on a large-scale click-through-rate model. Detaching six dense subarchitectures resulted in a training throughput improvement of up to 9.51%, all while maintaining accuracy close to the baseline. This method effectively utilizes spare capacity, leading to faster training times for complex recommendation systems.