MeRLa Improves RLHF Alignment with Meta-Learned Reward Shaping

Reinforcement Learning from Human Feedback (RLHF) is the standard method for aligning large language models (LLMs) with human preferences. However, its effectiveness is often limited by static, task-agnostic reward models, which can lead to sparse learning signals and suboptimal alignment. This new research presents MeRLa (Meta-Learned Reward Shaping), a principled framework designed to overcome these limitations. MeRLa meta-learns a task-aware shaping function across auxiliary tasks before the main RLHF training. This learned function generates a composite reward that not only preserves policy optimality but also provides rich, task-specific learning signals. The meta-objective combines task discrimination, entropy regularization, and potential-based conservation to ensure stable convergence. Experiments conducted on LLaMA-3-8B across four benchmarks demonstrated that MeRLa consistently outperforms existing methods like PPO, DPO, GRPO, and DAPO, achieving a 90.8% length-controlled win rate on AlpacaEval 2.0 and significantly improved stability during training.