Fora Protects LLM Capabilities During Fine-Tuning

A new fine-tuning technique called FORA (Function-space Orthogonal Residual Adaptation) has been developed to prevent large language models (LLMs) from losing previously acquired capabilities when adapted to new tasks. Unlike existing methods that rely on proxies like parameter distances or weight singular directions, FORA directly identifies and protects the specific activation subspaces within the model that are essential for a given capability. This is achieved by estimating the principal directions of input-activation covariance per layer using label-free calibration inputs. FORA constructs a high-capacity branch for new task learning that is structurally barred from interfering with capability-relevant function directions, complemented by a narrow spectral channel for controlled plasticity. This construction can also be extended for parameter-efficient adaptation. Across various tasks, including preserving translation while learning COGS and GSM8K, FORA consistently demonstrated superior capability preservation compared to weight-space projection and standard regularization, with only a minor trade-off in new-task performance in some settings. The key advantage stems from projecting onto capability-derived directions rather than weight-derived ones.