VPOS: Faster, More Accurate Feature Selection for Machine Learning

A new greedy unsupervised feature selection framework, Variance-Preserving Orthogonal Selection (VPOS), has been developed to enhance the efficiency and accuracy of identifying crucial features in datasets. This method operates within the weighted PCA loading space, employing an orthogonal deflation technique. After each feature selection, VPOS projects out the chosen feature's variance direction, ensuring subsequent selections cover distinct, orthogonal aspects of the data's covariance structure. This approach guarantees a reduction in the loading matrix rank with each step. The VPOS framework's objective is linked to monotone submodular maximization, and its single hyperparameter is determined by minimizing reconstruction mean squared error. Extensive evaluations across eight benchmark datasets demonstrate VPOS's superior performance. It consistently achieves the lowest reconstruction MSE on all benchmarks, while also exhibiting a remarkable speed advantage, running 10 to 140 times faster than comparable graph-based methods at scale. The core innovation of deflation is confirmed to be the primary driver of its success, leading to a 10-73% reduction in MSE compared to traditional PCA without deflation.