Meta-LoRA Improves LLM Personalization Across Domains with Sparse Data.

A new method called PAC-Bayes-regularized Meta-LoRA has been introduced to address the challenge of personalizing large language models for new conversational domains with limited user interaction data. Existing personalization techniques often struggle with overfitting when evidence is sparse or can transfer irrelevant information from source domains, leading to unreliable results. Meta-LoRA tackles this by using a meta-learned LoRA initialization as both a starting point and a prior, dynamically adjusting the strength of model updates based on the amount and certainty of available evidence. This approach prevents overfitting with minimal data while allowing for stronger personalization as more evidence becomes available. The method also functionally separates personalization priors into stable user preferences and domain-specific hidden-space conditioning. Experiments across multiple benchmarks, including HiCUPID, show significant improvements, reducing cross-domain win-rate degradation by 47.9% and boosting win rates by 110.2% in cold-start scenarios for unseen users.