New Benchmark Reveals Label Noise Challenges for ML Models.

Evaluating machine learning methods designed to handle noisy labels often relies on synthetic instance-dependent label noise (IDN) benchmarks. However, existing methods for generating this noise, typically through simulated imperfect annotators or classifiers, often leave the underlying source of ambiguity unclear. This research presents CILN, a novel framework for generating IDN benchmarks by introducing controlled input corruptions. A diverse group of "voters" then labels these corrupted instances, resulting in datasets where both the origin and intensity of ambiguity are explicit and manageable. The framework was used to create 90 benchmark settings across CIFAR10, MNIST, and Adult datasets, covering various corruption types and severity levels. Experiments confirmed that CILN-generated benchmarks exhibit genuine instance-dependent noise and diverse confusion structures. On CIFAR-10, these benchmarks produced label distributions closer to human uncertainty than existing synthetic IDN benchmarks. Crucially, corruption-mediated IDN exposed failure modes in popular noisy-label learning methods like Co-Teaching and DivideMix that were not apparent under comparable levels of rater-fallibility noise. This highlights that the specific structure of noise, beyond just its rate, plays a significant role in benchmark difficulty and algorithm behavior, offering a complementary framework for studying noisy-label learning.