Monitoring Stability in Continual Personalization of Small Language Models

Small Language Models (SLMs) are increasingly being considered for deployment on edge devices, enabling personalized applications with low latency and enhanced privacy. However, personalizing these models over time requires continual adaptation to evolving user data, which introduces the risk of catastrophic forgetting—where learning new information degrades performance on previously learned tasks or general capabilities. Previous benchmarks have shown that continual fine-tuning can significantly harm the general abilities of aligned large language models. This research presents a detailed study on sequential LoRA personalization for SLMs. It employs a checkpoint-level protocol, saving model states after each adaptation stage and evaluating them against current tasks, previously seen tasks, and a fixed reference set. This monitoring approach allows for tracking task performance, forgetting, and reference set drift over time. The study reveals that lightweight reference set distributional diagnostics can uncover model-specific instability patterns during sequential LoRA personalization, even when task-level metrics appear stable. This highlights new avenues for robust stability monitoring in continual learning settings for SLMs.