LoRA Finetuning Memory Reduction for Edge LLMs

This research addresses the significant memory constraints encountered when fine-tuning large language models (LLMs) using Low-Rank Adaptation (LoRA) on edge devices. While LoRA enables personalized AI experiences and data privacy, the peak memory requirements often exceed the capabilities of consumer hardware, especially for large models and long-context training. The paper proposes a comprehensive set of complementary techniques designed to reduce memory footprint without compromising model quality. These include quantizing the base model with on-the-fly dequantization, implementing memory-efficient checkpointing through selective activation caching and disk offloading, approximating softmax computations using semantically relevant token subsets, and applying logits masking. Experimental evaluations on Llama-3.2 3B and Qwen-2.5 3B models demonstrate substantial memory reductions, achieving up to 26x and 28x peak memory savings, respectively. These advancements make it feasible to perform LLM fine-tuning directly on resource-limited edge devices, enhancing personalization and data privacy.