Von-Neumann Transformer Boosts Neural Decoding Efficiency

A new neural decoding model, the Von-Neumann State-Space Transformer (VN-SST), has been developed to improve sample efficiency, particularly in scenarios with limited data and computational resources. Unlike standard Transformers that apply a uniform operator across all tokens, VN-SST adopts a novel architecture inspired by von-Neumann principles. It employs a low-rank "instruction bank" where a shared base operator is augmented by a small set of learned low-rank instructions. A per-token code, derived from a low-dimensional state-space memory, then synthesizes the specific weight matrix used for that token. This design mimics how low-dimensional dynamics might route cortical computation. Evaluations on three motor-cortex neural-decoding benchmarks show VN-SST significantly outperforms modern Transformers in data efficiency, especially with scarce data. It also maintains or improves accuracy with longer context lengths, a common challenge for other models. Furthermore, the model demonstrates parameter efficiency on small text benchmarks, suggesting its underlying mechanism could be broadly applicable beyond neural decoding.