New PEFT Method Learns Optimal Adaptation Domain for LLMs.

A novel parameter-efficient fine-tuning (PEFT) technique, named Fractional-Fourier Mixture of Experts (FRAME), has been developed to enhance the adaptation of large language models. Unlike existing methods that fix the adaptation domain to either spatial or Fourier, FRAME introduces a learnable fractional-Fourier order for each expert in a mixture-of-experts adapter. This allows the model to continuously interpolate between spatial and Fourier domains, selecting the most suitable basis for each low-rank update. The core innovation lies in treating the choice of domain as a trainable parameter. By enabling experts to operate across a spatial-spectral continuum, FRAME ensures that each low-rank update is applied in the most compact and effective domain. This dynamic domain selection, coupled with the inherent decorrelation of fractional-Fourier operators, significantly reduces interference between experts and improves multi-task composition. Evaluations on LLaMA-3.1-8B and Qwen2.5-7B across diverse benchmarks (commonsense, mathematical, code, knowledge) demonstrate that FRAME outperforms strong MoE-LoRA and spectral baselines like FlyLoRA and FourierMoE. The method adds negligible computational cost and reveals that learned orders specialize interpretably by task and layer, indicating its effectiveness and adaptability.