Mixture-of-Control Enhances Transformer Fine-Tuning Efficiency

This research introduces Mixture-of-Control (MoC), a novel and efficient fine-tuning framework for transformer-based models. Traditional state-based fine-tuning methods, which update lightweight controls into states rather than model weights, offer memory savings but often limit information exchange between different transformer blocks by applying only per-block updates. While some prior mechanisms allow cross-block communication, they typically incur significant computational overhead.MoC addresses these limitations by adaptively combining local and global control signals to improve representation learning. It conceptualizes block-wise control states as "experts" within a sparse mixture-of-experts (MoE) process. This design facilitates efficient communication across transformer blocks without introducing the substantial computational burden seen in other cross-block communication mechanisms.Empirical evaluations across various transformer-based benchmarks demonstrate that MoC consistently surpasses existing state-based fine-tuning methods. Crucially, it achieves this superior performance while maintaining comparable memory and computational efficiency, making it a practical and powerful alternative for adapting large transformer models.