MARL Reward Attribution: Impact on Agent Behavior and Representation Geometry.

In cooperative multi-agent reinforcement learning (MARL) systems, it's common practice to use team-averaged rewards, meaning each agent receives feedback based on the overall team outcome, irrespective of its individual contribution. This research explores whether this choice of feedback attribution leaves a discernible impact, either geometrically in the learned representations or behaviorally, on the agents. The study proposes two low-overhead diagnostic metrics: EffRank/$n$ (effective rank normalized by agent count) and $D_\text{act}$ (mean pairwise KL divergence between agents' action distributions). These metrics are designed to help identify the effects of different reward attribution schemes. The researchers tested these diagnostics on competent MAPPO agents within the SMACv2 protoss_5_vs_5 environment, where unit type information is embedded in the observation. The findings indicate that the geometry of learned representations is more influenced by the observation of unit type than by the reward attribution method. When unit type was observed, both shared and individual rewards resulted in similar EffRank/$n$ and probe accuracy. However, $D_\text{act}$ showed higher values under individual rewards, suggesting behavioral differences. Masking unit type significantly reduced the probe signal, highlighting the importance of observed role information. The conclusion is that while individually rewarded agents are competent and role-separable, observed role information primarily explains representation geometry, with reward attribution mainly affecting behavior. The proposed metrics, EffRank/$n$ and $D_\text{act}$, are valuable for diagnosing reward attribution effects, especially when controlling for observed role information.