Credit Scoring Reject Inference Creates Illusion of Improvement

Reject inference methods are widely employed in credit scoring to mitigate survival bias, which arises because only approved applicants' performance is observed. However, the true effectiveness of these methods has been poorly understood. This study systematically evaluates several such strategies and uncovers a critical flaw. The research identifies a structural failure mode where, during natural retraining cycles, credit scoring models can appear to improve in accuracy while their ability to correctly screen out defaulters (rejection quality) significantly deteriorates. This creates an "illusion of improvement," leading practitioners to mistakenly believe their systems are getting better. To counteract this, the study proposes a controlled exploration strategy. This involves deliberately approving a small fraction of rejected applicants to observe their actual outcomes, thereby breaking the problematic feedback loop without relying on statistical assumptions. Experiments show that even minimal exploration rates (2-5%) are sufficient to diagnose the severity of this feedback loop at negligible cost, confirming that standard evaluation metrics are inadequate under selection bias.