Research Mitigates Early Training Collapse in CTR Models

Deep learning models used for predicting click-through rates frequently encounter a phenomenon known as "early training collapse." This occurs when the model's performance on validation data sharply declines after the initial training epoch, even as the training loss continues to decrease. This instability hinders effective learning and limits the overall efficacy of the model. Researchers investigated this issue using large-scale industrial datasets to understand its root causes and evaluate potential solutions. Their analysis revealed that while adjusting the learning rate offered only minor improvements, managing feature sparsity yielded substantial benefits. By either removing features that are excessively sparse or aggregating infrequent feature values, they were able to stabilize the training process. This approach allowed models to continue learning effectively beyond the first epoch, leading to better performance in both offline evaluations and live online systems.