SHiPPO Enhances Recurrent Memory for Selective State Space Models.

The HiPPO (High-order Polynomial Projection Operator) method provides recurrent states with memory semantics by representing them as coefficients of online polynomial projections, but within fixed channel coordinates. In contrast, contemporary selective State Space Models (SSMs) rely on token-dependent control and dynamic channel interactions. This research bridges this gap by introducing SHiPPO, or Sylvester HiPPO. SHiPPO is a transported projection-memory prior that elevates HiPPO's coefficient memories into a moving channel frame. This allows the approximation family and channel metric to be transported together along a specified path. Conditional on this path, the state behaves like an ordinary HiPPO in a tied moving frame, following Sylvester coefficient dynamics. This design preserves the left online-memory operator while incorporating right-action transport, which is vital for selective-SSM execution. The authors derive a restricted, group-local realization of SHiPPO compatible with controllers, featuring exponential-adjusted updates, exact block-affine scanning, and recurrent decoding. Diagnostic tests show that while increasing current-token write rank improves prediction error, it struggles to recover order-sensitive changes to previously written memory. SHiPPO's transported-memory variants, however, successfully recover this signal, demonstrating its superior ability to handle dynamic memory updates. This positions SHiPPO as a mechanistically grounded transported-memory prior, particularly beneficial for complex sequence modeling tasks requiring nuanced memory handling.