New RL Approach Boosts Efficiency with Deep Critics

Recent advancements in offline reinforcement learning (RL) have often relied on complex generative actors, such as diffusion or flow-matching policies, which can capture diverse behaviors but incur substantial computational overhead during deployment. This research re-evaluates the allocation of model capacity in offline actor-critic methods, arguing for investing more capacity in the critic, which is only used during training, rather than the actor, which runs at every decision step. Historically, scaling MLP critics in offline RL has led to instabilities. The authors identify three specific failure modes: optimization challenges, amplification of bootstrap noise, and value-range drift. They address each of these with corresponding solutions: a residual MLP backbone for optimization, n-step bootstrap targets to mitigate noise, and a categorical cross-entropy loss to stabilize value ranges. By combining these ingredients with a lightweight deterministic actor, they introduce LAC (Light Actor, deep Critic). On the OGBench benchmark, LAC achieves performance comparable to the most powerful diffusion and flow-matching baselines, but with up to four times lower inference latency. This efficiency is similar to one-step distilled policies, without the need for distillation, demonstrating a scalable path for offline RL.