Structured RL Optimizes Bayesian Persuasion for Interactive Driving

Dynamic traffic management can be significantly enhanced by intelligent lead vehicles that coordinate the route choices of connected vehicles using real-time data. This scenario presents a challenge in harmonizing decisions, which the paper addresses through the strategic information-revealing framework of Bayesian persuasion. Here, a principal (the lead vehicle) aims to influence an agent's (connected vehicle's) sequential decision-making by selectively disclosing information, such as upcoming traffic conditions, through carefully designed signals. The complexity arises because the agent's responses are farsighted, maximizing its own long-term reward, making the principal's signaling strategy computationally intensive to design. To overcome this computational hurdle, the researchers introduce an online structured reinforcement learning framework. This framework is designed to synthesize computationally efficient signaling strategies that are persuasive for a farsighted agent. The paper's contributions include MAPL, a structured policy learning algorithm for faster online learning with monotonic agents, and the identification of sufficient conditions for the supermodular structure of the principal's Q-function. Furthermore, the research identifies conditions to ensure the persuasiveness of the principal's signaling strategy and proposes Supermodular Q learning for Principal (SQP), which leverages this supermodular structure to create efficient and persuasive strategies for monotonic learning agents. Numerical analysis, applied to a real-time Bayesian persuasive driving scenario for lane selection, demonstrates that the proposed method is 30% more cost-efficient in optimizing travel rewards for both the lead and connected vehicles compared to existing signaling strategy design methodologies.