Conformal Prediction Fails Minority Class in Drug Discovery

This research uncovers a significant flaw in the application of standard conformal prediction methods within drug discovery, particularly when dealing with imbalanced datasets. While conformal prediction is designed to provide reliable uncertainty estimates by guaranteeing a certain coverage rate for predictions, the study demonstrates that this guarantee often holds only marginally, meaning globally, but fails catastrophically for minority classes. For instance, on datasets like clinical-trial toxicity, minority class coverage plummeted to as low as 4.2%, despite an overall 90% coverage target being met. The failure is not tied to a specific model architecture, appearing consistently across random forests, graph networks, and chemical language models. An underlying conservation identity explains this effect: the shortfall in the minority class's coverage is directly proportional to the majority's surplus, amplified by the imbalance ratio. This makes the problem both severe and easily overlooked, as aggregate metrics remain reassuringly high. The paper proposes a practical solution: class-conditional, or Mondrian, conformal prediction. This approach successfully closes the coverage gap on all tested datasets, restoring the minority class to its target reliability with only a modest increase in prediction-set size. The findings highlight the importance of per-class reliability in critical applications like virtual screening, where abstaining on affected compounds can transform a net-negative screening campaign into a net-positive one.