p-Spin Glass Network Enables Efficient Continual Learning

A new neural network architecture, the p-Spin Glass Network, has been developed to address significant limitations in modern sequence models, particularly their reliance on large memory footprints and large-batch stochastic optimization. These traditional requirements hinder sample efficiency and the ability to perform continual learning effectively. The p-Spin Glass Network introduces several key capabilities. It enforces memory efficiency by using native ternary quantization, which compresses internal parameters by eight times, and by strictly bounding activation memory. Furthermore, it exhibits remarkable sample efficiency, achieving performance comparable to Transformer baselines while requiring eight times fewer training sequences. Crucially, this architecture enables stable and smoothly converging single-batch learning, even with a micro-batch size of one. This stability is modality-agnostic, maintaining robust temporal credit assignment across diverse data types, from discrete subword tokens to long, uncompressed raw byte streams. This breakthrough removes the necessity for large batches in stable deep learning, paving the way for more practical continual learning and edge AI applications.