SSM Adapters Outperform LoRA for Long-Context Fine-Tuning

Parameter-efficient fine-tuning (PEFT) methods typically focus on attention projectors, but their effectiveness for tasks requiring sequential state accumulation, especially with long contexts, remains underexplored. This study investigates whether PEFT can benefit from State Space Model (SSM) adapters and if MLP blocks are better injection sites. The researchers introduce the Hankel Reduced-order Model (HRM) adapter, an SSM-based residual module initialized using Balanced Truncation of empirical Hankel Grammians. HRM adapters leverage the time-invariance of the system matrix to enable an exact FFT-based parallel scan, achieving computational efficiency comparable to LoRA across all context lengths. In evaluations on Mistral-7B with iso-parametric settings, HRM adapters consistently outperformed LoRA variants on LongBench tasks, showing substantial relative accuracy improvements (e.g., +34.8% on QuALITY, +71.6% on QMSum). Further analysis revealed that HRM adapters effectively learn to modulate recurrence, positioning them as a robust architectural alternative to low-rank adaptation for long-context sequence modeling.