Conditioned Direct Feedback Alignment Improves Neural Network Training

Researchers have investigated Direct Feedback Alignment (DFA), an alternative to backpropagation for training neural networks, focusing on its limitations and potential improvements. DFA uses fixed random projections of output error to train hidden layers, bypassing the need for a transposed-weight backward pass. The study identifies that anisotropy in either the presynaptic-activity or local-error factors during DFA's outer-product weight update can lead to training failures. The paper demonstrates that conditioning these factors can significantly enhance DFA's performance. Specifically, activity conditioning showed substantial gains when high-variance directions contained irrelevant noise, while error conditioning also improved raw DFA. Combining both independently selected activity and error factors yielded further, albeit smaller, improvements. This work introduces a symmetric block-local family of normalized DFA (nDFA) methods, including activity nDFA and error nDFA, which apply inverse second moments for preconditioning. While the error factor proved fragile when under-damped and BatchNorm emerged as a strong activity-side alternative, the research frames conditioned DFA as a detailed study into why local outer-product rules can fail, rather than a universal replacement for backpropagation or a complete solution for convolutional credit assignment.