Bounding Causal Impact of ML Decisions Using Counterfactual Correctness.

Machine learning models are increasingly used to assist human decision-making in critical areas like healthcare and criminal justice. Evaluating the true causal impact of these systems on outcomes, such as patient survival or crime rates, is essential. While Randomized Control Trials (RCTs) provide strong evidence, running new trials every time a model is updated or retrained is often impractical. This work introduces a partial-identification approach that uses existing RCT data to establish bounds on the causal effect of a new model. The core innovation lies in incorporating assumptions that link fine-grained predictive accuracy to downstream outcomes. Specifically, it uses two monotonicity assumptions: one on individual-level "counterfactual correctness" (a correct prediction leads to non-inferior outcomes) and another on the relationship between subgroup predictive performance and outcomes, interpreted as trust in model outputs. A simulation study demonstrates that this method yields more informative causal bounds compared to previous approaches by effectively integrating this additional information. This offers a practical way to assess the impact of ML model updates without requiring entirely new, costly RCTs.