Sequence-Pooled Normalization Provides Global Context Beyond Receptive Fields

Traditional understanding suggests that a convolutional sequence labeler's receptive field defines the maximum context it can utilize. However, new research demonstrates that normalization layers, by computing statistics from the entire input sequence during inference, create a global information path. This path effectively bypasses the local constraints of the receptive field, allowing the model to access broader context. The study mathematically derives this effect and shows its substantial impact on performance, particularly in tasks where labels persist over long sequences. The findings indicate that this global context from normalization can account for nearly all the benefits typically attributed to larger receptive fields in such scenarios. Furthermore, the research reveals that standard attribution methods, like ablating receptive-field-enlarging blocks, can significantly overstate their contribution because they inadvertently sever this normalization-induced global path. This suggests a need to re-evaluate how context is understood and attributed in sequence labeling models.