New Method Speeds LLM Adaptation with Forward-Pass-Only Training

Researchers have introduced a novel technique called Forward-Pass-Only (FPO) MLP training, designed to adapt large language models (LLMs) more efficiently. This method bypasses the need for a backward pass through the main body of the model, which traditionally consumes significant computational resources. By doing so, FPO achieves a 2.7 to 3.2 times higher throughput and approximately 40% less peak training memory compared to conventional fine-tuning approaches. The efficacy of FPO stems from an empirical observation: the prediction error at the output layer of a transformer model closely approximates the true gradient in later layers. FPO leverages this by computing a single error signal at the output and applying it directly to target layers, without propagating signals between layers or constructing an autograd graph. Evaluations across models like OLMo-2-7B, Qwen3-8B, and Falcon3-7B demonstrate that FPO improves in-domain perplexity while maintaining off-domain benchmark performance within baseline noise levels, a consistency often lacking in full-network fine-tuning.