MetaNCA Learns Rules for Neural Network Self-Organization

Inspired by biological self-organization, where individual components interact locally to drive emergent properties, researchers have developed Meta Neural Cellular Automata (MetaNCA). This framework learns local rules that enable the self-organization of weights within artificial neural networks. Unlike traditional methods, MetaNCA allows a learned rule network to iteratively update the weights of a "task network" using only local interactions on the computation graph. A key innovation is the proposed Weight Transformer architecture for the local rule network, which employs linear attention to aggregate signals from neighboring weights and hidden states. Once trained, this rule network can generate task networks with diverse architectures, such as feedforward MLPs, CNNs, and ResNets, without requiring backpropagation for each new architecture. The MetaNCA framework has been successfully demonstrated on MNIST and CIFAR-100 datasets, scaling to networks with up to 2 million parameters. Crucially, it shows strong generalization capabilities, producing weights for architectures not encountered during its meta-training phase. The study also indicates that introducing architectural diversity during the training phase further strengthens this generalization ability.