New Method Corrects Noisy Human Labels for AI Classifiers

Researchers frequently employ automated classifiers to label unstructured data for statistical analysis. While existing methods can rectify errors in these automated labels using a probability-sampled audit set, they typically assume the human audit labels are perfectly correct. In reality, human labels often contain noise, and only a subset of audited items might undergo expert review or adjudication. The proposed method, Partially Adjudicated Design-Based Supervised Learning (PA-DSL), addresses this challenge. It leverages expert-adjudicated cases to first correct the noisy human labels. Subsequently, this refined audit information is used to debias analyses derived from the complete set of automated labels. The estimator is designed to be valid for a broad range of downstream analyses, provided that the audit and adjudication probabilities are known. Experiments, including synthetic and semi-synthetic scenarios, demonstrate that PA-DSL maintains nominal coverage and significantly reduces RMSE compared to using only adjudicated labels, especially when noisy human labels still contain recoverable signal.