Dynamical Systems Theory Explains LLM Response Classification

Recent empirical work has shown success in classifying large language model (LLM) responses by treating token embeddings as trajectories within a black-box dynamical system and comparing prediction residuals. However, a theoretical understanding of this approach, including its scalability and transferability across different embedding models, has been lacking. This paper addresses these gaps by formalizing the classification task as a binary hypothesis test between two stochastic linear dynamical systems. The study reveals that while the total variation distance between the stationary marginal distributions of two dynamical systems can be small even with substantial dynamic differences, ignoring token dynamics can lead to a fundamental accuracy floor for classifiers. Crucially, the misclassification probability of dynamical system-based classification decreases exponentially with the sequence length, governed by a "dynamical discriminability" quantity that measures the spectral distance between the systems. Furthermore, the research characterizes cross-embedding generalization by introducing an approximate intertwining condition between embedding models. It establishes a lower bound on transferable discriminability based on the intertwining map's smallest singular value. These findings collectively provide a theoretical basis for the observed empirical performance of dynamical system-based LLM classification and advocate for further application of dynamical system theory in analyzing AI systems.