Factorized Spectral Representations Enhance Reinforcement Learning Efficiency

Deep reinforcement learning often struggles with sample efficiency, requiring vast amounts of interaction data to build effective world models. A common approach involves spectral representation methods, which view the transition process as a matrix and use self-supervised contrastive objectives to learn low-rank factorizations. This new research extends this concept by considering the transition kernel as a three-mode tensor, encompassing states, actions, and next states. The proposed method, FaStR (Factorized Spectral Representations), utilizes a CP decomposition on this tensor, coupled with a noise contrastive objective. This decomposition yields distinct encoders for states, actions, and next states, which collectively form a unified spectral representation. This factored structure leads to a more constrained hypothesis space, reducing the required sample size for representation learning by a factor proportional to the smaller of the state or action dimensions. Empirical results demonstrate that FaStR provides substantial improvements, particularly in high-dimensional locomotion tasks where the underlying dynamics align well with the factored structure. A notable advantage is the transferability of the learned state encoder across different actuator configurations, requiring only the action encoder to be retrained.