Safety Alignment in LLMs Localized to MLP Layers

The safety alignment mechanisms within large language models (LLMs) are often perceived as a property distributed across the entire network. However, the observed fragility of these safety features suggests that refusal behavior might be concentrated in a more specific set of parameters. This study investigates the precise location where safety-aligned refusal is encoded by selectively transferring weights from aligned models to unaligned base models. Experiments conducted across two open-weight model families and four safety benchmarks consistently showed that MLP (Multi-Layer Perceptron) weights are the dominant factor in transferring refusal capabilities. Replacing MLP parameters recovered significantly more malicious-prompt refusal than replacing attention parameters. Furthermore, within the MLP stack, the parameters responsible for refusal behavior exhibited a clear concentration in the mid-network layers, specifically blocks spanning layers 8-11. The findings also indicate that the composition of these safety-relevant components is non-additive, meaning that simply adding more aligned blocks can sometimes reduce refusal performance, suggesting a complex interplay of parameters.