Exact Dimensionality Reduction for Non-Smooth Stochastic Complexity and Sampling.

Calculating the Normalized Maximum Likelihood (NML) codelength for non-smooth estimators, such as Lasso, has historically been hampered by high computational costs. Specifically, the geometric Propose-and-Project Metropolis-Hastings (PPMH) sampler requires inverting large matrices and computing determinants, leading to cubic scaling walls (O(N^3)) at each step. This research presents a groundbreaking, exact mathematical reformulation that circumvents these bottlenecks. By leveraging the block Schur complement and Sylvester's determinant identity, the computational complexity for both projection operator evaluation and volume factor calculation is drastically reduced. The new method achieves a complexity of O(k^3 + N^2k) per step, where k is typically much smaller than N. The dimensionality reduction is generalized to other models like Sparse Support Vector Machines, Elastic Net, and Group Lasso. A rigorous numerical stability analysis is provided, and empirical benchmarks on high-dimensional datasets confirm a constant speedup exceeding 14,100 times, while maintaining double-precision numerical equivalence. This makes exact non-smooth NML estimation highly tractable for large-scale statistical inference, opening new possibilities for complex model analysis.