Deep RL Agents Develop Sparse Representations with Frozen CNNs

A surprising phenomenon has been observed in deep reinforcement learning (DRL): agents trained using frozen, randomly initialized Convolutional Neural Network (CNN) feature extractors spontaneously develop highly sparse representations in their subsequent fully-connected layers. This sparsity emerges without any explicit objective to induce it. For instance, in a game like Pong, agents might compress task-relevant information into as few as 1-3 neurons out of 64, whereas trainable CNNs would activate many more. The study establishes several key findings: the degree of sparsity correlates with task complexity, with simpler tasks exhibiting greater sparsity. Within the same game, different training seeds can lead to varying levels of sparsity, which in turn impacts achievable performance. Ablation studies confirm that these active neurons are essential for the agent's performance. Furthermore, the critical set of active neurons locks in early during training, long before the agent achieves positive reward. This suggests that gradient descent on a frozen random projection can reveal the effective rank or intrinsic dimensionality of a problem, even without specific sparsity mechanisms.