Hyper-ES Boosts LLM Reasoning with Efficient Evolution Strategies

Large Language Models (LLMs) often struggle with efficient reasoning, especially when resource constraints limit gradient-based fine-tuning. Evolution Strategies (ES) offer an alternative, but their direct application to LLMs with billions of parameters is inefficient due to the vast, high-dimensional parameter space where random perturbations are unlikely to yield useful updates. Hyper-ES addresses this by introducing a subspace-based ES framework. Instead of random exploration, it first identifies a small number of effective descent directions through inexpensive gradient-based fine-tuning runs. These directions, though individually limited, collectively form a compact "adaptation subspace" that captures meaningful reasoning updates. Within this subspace, Hyper-ES employs CMA-ES to optimize layer-wise merging coefficients, allowing the system to search over combinations of these pre-identified meaningful directions. This approach avoids the pitfalls of full-parameter random search while leveraging ES's strengths in lower-dimensional optimization. Experiments on Qwen2.5-Instruct and DeepSeek-R1-Distill across mathematical reasoning datasets demonstrate that Hyper-ES consistently outperforms existing methods like GRPO-LoRA, achieving better results with fewer gradient updates.