Estimating Classifier Uncertainty: Improved Confidence Intervals for LLMs.

Researchers often use text classifiers, including large language models, to quantify constructs from natural language, reporting metrics like recall and precision. However, measures of uncertainty, such as confidence intervals, are frequently omitted or calculated using inappropriate methods, especially with small datasets, high performance, or data nested within individuals. This paper rigorously evaluates various confidence interval methods under conditions typical for social science text classification. It reveals that standard approaches like the Wald interval and basic percentile bootstrap often yield inaccurate results, with coverage significantly below the nominal 95% level. The study recommends more accurate methods, including Agresti-Coull, Wilson, Clopper-Pearson, and introduces a novel pseudo-count regularized bootstrap, particularly useful for F1 score calculation. For nested data structures, it emphasizes the necessity of adjusting for effective sample size and degrees of freedom to produce reliable analytic intervals. This guidance aims to enhance the transparency and validity of machine learning applications by promoting better uncertainty reporting.