New Zero-Order Optimization Boosts LLM Fine-tuning Efficiency.

Fine-tuning large language models (LLMs) is crucial for adapting them to specific tasks and data, but it faces significant challenges, particularly the high memory overhead associated with backpropagation. This process requires storing activations, gradients, and optimizer states, which can be prohibitive for very large models. Zero-order (ZO) optimization offers a memory-efficient alternative by not requiring gradient computations. However, ZO methods are highly sensitive to hyperparameters like step size and smoothing parameters, often necessitating extensive and costly task-specific tuning. Parameter-free (PF) optimization aims to overcome this by adapting algorithmic parameters without prior knowledge of problem-dependent constants. This work introduces AdaNAGED, a new method that combines gradient-free training, adaptive parameter tuning, and non-Euclidean update geometry, which is beneficial for accounting for the heterogeneous structure of parameter blocks in LLMs. AdaNAGED leverages linear minimization oracle (LMO)-based ZO optimization. The paper establishes convergence guarantees and validates the method's effectiveness on a large-scale LLM fine-tuning task using the OPT-1.3B model, demonstrating a novel approach for efficient fine-tuning.