ML Diabetes Risk Models Show Performance Gaps in Real-World Data

Machine learning models designed to predict Type 2 diabetes risk often show strong performance in initial internal tests. However, a new study conducting a large-scale external validation found that these models can lose significant effectiveness when applied to real-world, diverse populations. The research developed a multi-dimensional framework to assess discrimination, calibration, interpretability, and algorithmic fairness. An XGBoost model, trained on a national dataset using eight non-laboratory predictors like age, BMI, and smoking status, demonstrated good internal accuracy. Yet, when externally validated on a much larger, nationally representative population, its predictive accuracy decreased by nearly 10%. Crucially, the study highlighted significant fairness concerns: the model performed considerably worse for high-risk demographics, such as older adults and obese individuals, who are most in need of accurate predictions. It also tended to overestimate risk. These findings underscore the necessity for fairness-aware, age-stratified deployment strategies before such models are used clinically.