AHEAD Improves Multi-Class Label Aggregation with GNNs

Crowdsourced labeling is a vital source of data for machine learning, but it often suffers from noisy and biased annotations, especially in multi-class settings where individual annotators label only a small subset of tasks. Existing label aggregation methods struggle with accurately estimating annotator reliability under such sparse conditions. This research proposes AHEAD (cross-Annotator learning and High-confidEnce Annotator-guideD label aggregation), a novel framework designed to overcome these challenges. AHEAD utilizes a graph neural network (GNN) to learn high-dimensional "cross-annotator contexts," generating multi-view annotator embeddings that combine individual features with broader contextual information. These embeddings are then decoded into interpretable, annotator-specific confusion matrices, which are used to fit the observed labels. A composite objective, incorporating high-confidence annotators, helps stabilize the unsupervised training process. Experiments across 10 real-world datasets from various domains (NLP, CV, Video, Audio) show AHEAD substantially improving label accuracy, with average gains of over 4% and up to 14.9% in some cases, while also demonstrating scalability.