FCCA Boosts Low-Resource LLM Adaptation with Fewer Parameters

A new research paper presents Frozen-Core Cross-Covariance Adaptation (FCCA), an innovative parameter-efficient fine-tuning (PEFT) method designed for adapting large language models (LLMs) under extremely limited trainable parameter budgets. Unlike traditional PEFT approaches that focus on the quantity of parameters to update, FCCA emphasizes the strategic selection of *where* these limited parameters act. The method involves a calibration pass that fixes left and right bases for each weight matrix, with fine-tuning then optimizing only a small $r \times r$ core. FCCA estimates the signed input-error cross-covariance, whitens it using diagonal Fisher moments, and truncates it in a local metric to obtain stable core coordinates. This approach allows it to recover most of the benefits of more parameter-heavy methods like LoRA and DoRA, but with a drastically smaller trainable and optimizer-state cost. For instance, on Qwen2.5-3B, FCCA achieved an 83.0 macro-average, outperforming the next-best matched-budget constructor by 2.3 points and coming within 0.32 and 0.23 average points of LoRA and DoRA respectively, while optimizing only 36.9K parameters compared to their roughly 7.4M. The study highlights the critical importance of carefully selected fixed spans for effective low-resource LLM adaptation.