Uncertainty Sampling in Active Learning: Impact of Noisy Labels

Active learning strategies aim to reduce the cost of data labeling by intelligently selecting the most informative examples for annotation. Uncertainty sampling, a common approach, prioritizes examples where the model is least confident. However, these "uncertain" examples can also be the most challenging to label correctly, potentially introducing more errors into the training data. This research explores whether the failure of uncertainty sampling stems from acquiring more corrupted labels or if errors concentrated in difficult regions are particularly detrimental. The study compared margin-based uncertainty sampling with random sampling across three public binary tabular datasets, under conditions of clean labels, random classification noise (RCN), and difficulty-dependent noise. Various noise rates and annotation budgets were tested, using logistic regression as the model. Under clean label conditions, uncertainty sampling consistently improved balanced accuracy. However, when difficulty-dependent noise was introduced, the advantage of uncertainty sampling was reduced more significantly than with RCN on one dataset, but not on others. Exposure-matched analyses did not find universal evidence that structured error location imposes an additional penalty. The findings suggest that while uncertainty sampling can be label-efficient, its robustness is highly contingent on the specific dataset, the available labeling budget, the nature of the label noise, and the chosen evaluation metric.