LLM Explanations for Credit Risk Show Fidelity Issues

This research investigates the dual challenge of achieving high predictive accuracy and reliable explainability in credit scoring models, particularly when using large language models (LLMs) to generate decision rationales. The study developed a sophisticated multi-scale stacking ensemble model for credit risk prediction, which demonstrated superior performance, achieving a test ROC-AUC of 0.9539 and outperforming single models. However, the practical gain in avoiding missed defaults was operationally small. The central finding reveals a significant problem with the fidelity of LLM-generated explanations. In audited cases, the LLMs frequently misidentified the impact of features, sometimes stating factors were risk-increasing when attributions showed them as risk-reducing, or even inventing features not present in the input. This disconnect is attributed to inconsistencies in feature attribution methods (like SHAP and LIME) and issues with model calibration and perturbation stability. The paper concludes that while constrained prompting is helpful, post-generation verification of LLM explanations is essential, as fidelity cannot be assumed.