AuroOFT Enhances Quantized Fine-Tuning with Nonlinear Adaptations

This paper introduces AuroOFT, an advanced method for parameter-efficient adaptation of low-bit language models, building upon quantized orthogonal fine-tuning (qoft). While qoft effectively adapts models by learning structured activation rotations, its updates are limited to linear orthogonal transformations, restricting its ability to make input-dependent nonlinear corrections. AuroOFT addresses this by maintaining qoft as a stable, quantization-compatible branch while simultaneously attaching a zero-start gated low-rank nonlinear residual to each adapted linear layer. This residual maps activations into a compact latent space using adaptive nonlinear bases and token-dependent gating. Crucially, the zero-initialized up projection ensures AuroOFT is functionally identical to qoft at initialization. The results show that AuroOFT significantly improves Macro-6 scores over matched qoft (by 1.30-2.70%) and QLoRA (by 6.52-10.62%) on Qwen2.5 settings, while also saving 32.3-44.7% trainable parameters compared to QLoRA. This represents a substantial step forward in efficient and expressive fine-tuning for quantized models.