LoCA Enables Forward-Only LLM Tuning with One-Shot Calibration

A novel method called Local Credit Assignment (LoCA) has been introduced to streamline parameter-efficient post-training for large language models. Unlike traditional methods like LoRA that require repeated end-to-end backpropagation, LoCA amortizes the global credit assignment into a single, initial calibration step. This calibration involves a single backward pass to fit low-rank maps at each transformer block, linking final prediction errors to local hidden-state corrections. After this one-time calibration, LoCA reuses these maps to derive blockwise regression targets from forward activations, then fits low-rank adapters using closed-form ridge solves. This eliminates the need for any further backbone backward passes. The method demonstrates superior efficiency, achieving lower evaluation cross-entropy than LoRA in many comparisons, while significantly reducing GPU peak memory (26-29% lower), CPU steady-state memory (36-52% lower), and per-pass time (43-48% lower) after calibration.