Scalable Perturbation Learning for Online Self-Supervised ESNs

Developing intelligent systems that can adapt autonomously in real-world, high-dimensional environments is a key challenge. While self-supervised learning, online adaptation, and memory-efficient perturbation-based learning are desirable, they often conflict, especially in high-dimensional systems where perturbation-based methods suffer from variance that scales with the number of perturbed variables. This tension is particularly evident in Echo State Networks (ESNs) with large "reservoirs." This study introduces a novel perturbation-based learning rule specifically designed for online self-supervised learning in ESNs. The core innovation lies in an orthogonal decomposition of the self-supervised learning cost, which separates an input-dependent component from a redundant component tied to fixed ESN parameters. By perturbing only the input-dependent component, the effective perturbation dimension is drastically reduced from the large reservoir dimension to the much smaller input dimension. This approach successfully preserves the benefits of self-supervised adaptation, online learning, and scalar-feedback perturbation learning, while crucially avoiding the variance growth that typically plagues large ESNs. The findings suggest a new design principle for scalable and hardware-compatible learning, emphasizing that online learning should be restricted to the dynamically necessary low-dimensional components of the objective function.