TILR Improves LLM Reasoning Consistency and Stability

New research delves into the poorly understood structure of latent reasoning trajectories within language models, where multi-step inference occurs in the hidden-state space. The study reveals that the differences between strong and weak reasoning paths exhibit a concentrated low-rank structure, indicating the presence of stable, invariant directions amidst unstable, instance-specific variations. To leverage this insight, the researchers developed Trajectory-Invariant Latent Refinement (TILR), a novel training-free intervention framework. TILR operates by first learning a low-rank invariant subspace from the contrastive differences in reasoning trajectories across various inputs. It then constrains latent interventions to this stable subspace, while actively suppressing updates that are poorly aligned, using an adaptive alignment gate. Evaluated across six reasoning benchmarks, TILR demonstrated significant improvements. It enhanced answer consistency under paraphrases by roughly 10% and reduced latent trajectory variance by up to 50%, all while maintaining reasoning accuracy. These findings support a geometric interpretation of latent reasoning, suggesting that robust and transferable reasoning capabilities arise from stable, low-dimensional structures within the model's hidden states.