EMA-FS Accelerates GBDT Training with Gain-Informed Feature Screening

Gradient Boosted Decision Trees (GBDTs), such as LightGBM, spend a significant portion of their training time, often 65-70%, on constructing per-feature histograms. Existing methods for speeding this up, like random feature subsampling, discard features indiscriminately without considering their predictive value. Researchers have introduced EMA-based Feature Screening (EMA-FS) as a more intelligent optimization. EMA-FS maintains an exponential moving average (EMA) of the split gains for each feature across boosting iterations. After an initial warmup phase, it restricts histogram construction to only the top-K features, which are ranked by their historical gain. This gain-informed approach ensures that high-utility features are retained while less important ones are screened out, unlike random subsampling. The method operates at the per-tree level, ensuring full compatibility with LightGBM's existing histogram subtraction trick without requiring changes to core routines. Evaluations across diverse datasets, including financial fraud detection and advertising click-through prediction, demonstrated significant speedups. For instance, EMA-FS achieved a 2.61x speedup on a synthetic benchmark and 1.45x on the IEEE-CIS Fraud dataset. In some cases, it also improved AUC scores while delivering substantial speedups. A stochastic variant, S-EMA-FS, was also introduced, offering a unified framework for deterministic screening and random subsampling.