AQLoRA Speeds Up Quantized LoRA Fine-Tuning Without Search

Quantized fine-tuning (QLoRA) is a memory-efficient technique for large language models, but it often sacrifices training speed due to the on-the-fly dequantization of 4-bit weights. This research introduces AQLoRA (Adaptive-Quantization LoRA), a novel approach designed to mitigate this speed penalty. AQLoRA achieves faster training without the need for extensive search or calibration data. The core of AQLoRA involves a single CPU pass over the model weights. During this pass, layers are ranked by their NF4 reconstruction error. Under a specified memory budget, the top-K layers with the highest error are retained in fp16 precision, bypassing the dequantization step that slows down QLoRA. This adaptive selection process is entirely automated and requires no manual tuning or repeated calibration. Evaluations across various models and benchmarks demonstrated AQLoRA's effectiveness. The "speed setting" trained models 11.1% faster on average than well-tuned QLoRA, with only about a one-point accuracy reduction. A "quality setting" achieved 4.8% faster training while maintaining accuracy levels comparable to QLoRA and close to fp16 LoRA, for a modest increase in memory. The paper also provides crucial insights into reliable timing methodologies for shared hardware environments.