Model Merging Reveals New Jailbreak Risks in LLMs

Model merging is a technique that combines multiple finetuned language models without additional training, offering efficiency benefits. However, its implications for model safety have been largely unexplored. Previous assumptions often attributed merging risks solely to unsafe constituent models, implying that merging individually aligned models would preserve safety. Contrary to this, new research reveals a previously overlooked jailbreak risk that emerges specifically from the pretrained foundation model, even when all individual constituent models are safety-aligned. This means that the act of merging can expose latent vulnerabilities. To exploit this phenomenon, a new threat setting is proposed where an attacker can craft jailbreak prompts that generalize across different merged models sharing the same pretrained backbone, even without knowing the exact merging coefficients or constituent checkpoints. To facilitate this attack, a method called Basin-Aware Jailbreak (BAJ) has been developed. BAJ frames jailbreak generation as a min-max optimization problem over the merging space, designed to produce adversarial suffixes that are highly transferable across families of merged models. Experiments across various backbones and merging configurations demonstrate that BAJ consistently achieves high transfer success rates and remains effective even when existing defenses are in place, highlighting a critical new area for AI safety research.