Recurrent Network Redundancy Explored with Schur Coordinates

Recurrent neural networks often possess significant functional redundancy within their weight space. This means that certain modifications to a recurrent matrix might have little effect on the network's output for a given task, while other changes of similar magnitude could severely disrupt its behavior. Researchers explored this phenomenon in one-layer tanh RNNs by employing ordered real Schur coordinates. This mathematical approach helps to separate spectral blocks from directed nonnormal couplings, providing a diagnostic basis for targeted ablations that maintain the network's input and readout maps. The study found that the specific nonnormal Schur couplings that can be removed without significant performance degradation vary depending on the task and the trained solution. This suggests the existence of approximate functional invariances that are specific to the task rather than universal symmetries of the recurrent weight space, offering insights into how RNNs compute.