New Method Detects High-Dimensional Data Changes with Low-Rank Projection

Researchers have developed a new technique for identifying shifts in data distributions, particularly effective in high-dimensional settings where traditional methods struggle due to the lack of specified pre-change or post-change densities. The core of this approach involves projecting data densities onto a low-rank, degree-two space, which simplifies the complex task of density estimation into a more manageable problem of estimating matrix means. This transformation allows for robust detection of changes, even those related to data dependence rather than just mean shifts. The method, termed LRD, constructs a symmetric feature matrix from observations, enabling an isometric encoding of the density's degree-two orthogonal projection. By scanning matrix CUSUMs after rank truncation, it leverages the low-rank nature of the projected change. The technique has been shown to be computationally practical and accurate in experiments with high ambient dimensions, detecting subtle dependence changes that mean-based CUSUMs would miss.