Counterfactual Audits Reveal Toxic Mimicry in Medical RL for Sepsis

Offline reinforcement learning (RL) holds significant promise for optimizing critical medical decisions, particularly in intensive care units (ICU) for conditions like sepsis. However, current evaluation methods, such as Mean Squared Error (MSE) and Fitted Q-Evaluation (FQE), primarily assess how well an AI agent imitates observed behaviors. This limitation means they cannot identify a critical failure mode termed "Toxic Mimicry," where the AI might inadvertently learn and replicate harmful or inappropriate treatment patterns, such as withdrawing life-sustaining treatment during comfort-care transitions. To address this, a new framework called Counterfactual Clinical Audit (CCA) has been developed. CCA stress-tests RL agents by introducing physiological perturbations, aligning with established clinical guidelines like the Surviving Sepsis Campaign (SSC). Applying CCA to two prominent medical RL models, a Medical Decision Transformer (MedDT) and a Historical Causal Transformer (HCT-RL), revealed significant differences. The audit exposed that MedDT exhibited paradoxical behavior, reducing vasopressor dosage even as lactate levels escalated, directly contradicting standard resuscitation protocols. In contrast, HCT-RL, which incorporates causal action shielding and conservative Q-learning, maintained physiologically consistent responses. These findings underscore a fundamental disconnect between statistical fit and actual clinical safety, highlighting CCA as an essential new standard for evaluating medical RL systems.