RankShift Detects and Explains Categorical Data Shifts In-Database.

Many critical systems, such as login services or system logs, can experience significant shifts in the distribution of categorical events even when the total volume of events remains constant. For example, a specific error type might suddenly dominate logs without an increase in overall error rate. Traditional monitoring often misses these "categorical shifts." This research introduces RankShift, a method designed to detect and explain such changes directly within an analytical database. RankShift operates by comparing the share of each category within a current data window against a benign reference distribution using a Pearson score. This score not only signals a shift but also identifies the specific categories that are contributing most to the change, providing immediate explainability. The entire process—score calculation, alert calibration, and identification of contributing categories—is performed via a single database query. Evaluations on real-world datasets like HDFS, BGL, and Thunderbird logs show that RankShift performs comparably to or better than count-vector autoencoders, particularly in detecting rare-category shifts that are invisible to simple event-count monitoring. Crucially, RankShift requires no model training or inference service, making it significantly more lightweight and easier to deploy than machine learning alternatives, with a much smaller operational footprint.