Compliance2LoRA Enables On-Demand LLM Safety Alignment

This paper introduces Compliance2LoRA, a novel framework designed to address the growing need for personalized safety alignment in large reasoning models (LRMs). As LRMs become more customized for individual users, the demand for adherence to distinct subsets of safety policies creates a significant challenge. Traditional approaches, such as training a separate LRM for each policy subset, lead to prohibitive combinatorial overhead, while in-context learning methods incur high computational costs due to long context generation. Compliance2LoRA tackles this by employing a unified adaptive hypernetwork. This hypernetwork takes safety policies as customizable inputs and learns to generate specific LoRA (Low-Rank Adaptation) weights. When these generated LoRA adapters are applied to an LRM, they enable the model to produce responses that comply with the specified policy subsets. The research demonstrates that this hypernetwork-based training allows for on-demand policy adjustments on a single LRM. Crucially, it achieves this without sacrificing task performance across various model sizes and evaluation datasets, highlighting its effectiveness and practicality for adaptive, multi-policy compliance in LRMs.