Positional Encoding Affects Neural Network Symmetry Readout

When analyzing the weights of trained neural networks, especially those using positional encodings (PEs) in neural fields, researchers often try to uncover underlying geometric structures. This paper reveals that the symmetries detected from these weights are not solely the inherent symmetries of the data but are significantly shaped by the specific positional encoding used and the method of observation (readout observable). The study proposes an exact observability hierarchy, illustrating that even if a target function possesses a clear geometric symmetry, that symmetry might remain hidden at the weight level if the positional encoding cannot adequately represent the corresponding transformation. This implies a structural limitation imposed by PE design. Experiments with MLPs trained on 2D signed distance functions, using various shape symmetries, PEs, and Gram-based observables, confirmed this prediction. Different PEs consistently showed varying sensitivities to different symmetries, demonstrating that PE design impacts not only how well a network approximates a function but also which structural properties can be extracted from its learned parameters.