MM-XGBoost Enhances Robustness for Regression Tasks

XGBoost is a widely popular and powerful method for predictive modeling, known for its efficiency and scalability. It operates by iteratively fitting simple decision trees to the residuals from the previous step. While the standard quadratic loss function is commonly used, a Huber loss can also be applied. However, this research highlights a significant limitation: the performance of XGBoost can be adversely affected by the presence of vertical outliers and leverage points in the data. To address this vulnerability, the study explores alternative loss functions derived from robust regression techniques, specifically M-, S-, and tau-estimators. These estimators are designed to be less sensitive to extreme values in the dataset. The goal was to find a method that could maintain XGBoost's predictive power while significantly improving its resilience to problematic data points. The findings indicate that a two-step procedure, termed MM-XGBoost, provides the most effective solution. This approach achieves the optimal trade-off between robustness against outliers and overall prediction accuracy, making XGBoost a more reliable tool for regression tasks in real-world datasets that often contain noisy or anomalous data.