V-Simba Boosts Visual RL Efficiency in Continuous Control

Sample efficiency remains a significant hurdle in reinforcement learning (RL), particularly in visual RL where high-dimensional inputs complicate learning. While algorithmic advancements have been a focus, recent work in state-based RL suggests that architectural design alone can yield substantial gains. This paper introduces V-Simba, a visual RL architecture that translates these architectural principles to visual domains. V-Simba builds upon the Soft Actor-Critic (SAC) framework, integrating data augmentation, but its core innovation lies in architectural modifications: the addition of normalization layers for training stability and the use of pointwise convolutions for computational reduction. Despite its simplicity, V-Simba demonstrates performance comparable to or superior to state-of-the-art methods across challenging benchmarks like DMC, Adroit, and Meta-World, all while being more computationally efficient than existing solutions.