Sparse Bagging Framework Compresses Ensembles, Improves Calibration and Speed.

Standard ensemble learning methods, such as Random Forests or Bagged Neural Networks, typically assign equal voting power to all their constituent models. This uniform weighting can lead to overconfidence and ignores the varying local competence of individual estimators. This research introduces Simplex-Constrained Sparse Bagging (SCSB), a mathematically rigorous framework designed to address these limitations. SCSB focuses on post-training compression and probability calibration for bootstrap-based bagging ensembles. It reframes ensemble pruning and calibration as a joint optimization problem over the probability simplex, aiming to minimize the Out-Of-Bag (OOB) loss. To achieve sparsity and effectively prune less useful estimators, the framework introduces a concave quadratic penalty, overcoming the "L1-simplex paradox" where the L1 norm fails to induce sparsity on the simplex. The SCSB framework is model-agnostic and has demonstrated impressive results, achieving up to 96% ensemble compression. This significant compression translates directly into linear inference speedups and superior probability calibration, as measured by lowered Expected Calibration Error, all while maintaining or even enhancing generalization accuracy.