Hidden-State Optimization Order Improves Predictive Coding Networks

Local learning methods offer an alternative to traditional backpropagation for training deep neural networks, but they often struggle to learn robust features effectively. This research investigates whether the order in which hidden states are optimized can address this limitation, specifically within predictive coding networks (PCNs). PCNs are a local-learning framework where both hidden activities and prediction errors are explicitly managed during inference. The study proposes a "boundary-first" inference schedule. This schedule divides the model into chunks, first optimizing hidden states at the boundaries between these chunks, and then refining the representations within each chunk. When applied to PCNs, this boundary-first approach significantly boosted accuracy on the CIFAR-10 dataset, showing improvements of 9.77% and 5.51% under different parametrizations. Diagnostic analyses confirmed that this method led to more meaningful updates in early layers, greater diversity in layer-wise gradients, and stronger overall feature learning, suggesting it's a valuable design principle for local-learning systems.