Ground Truth Datasets Are Human Constructions, Not Objective

This paper posits that "ground truth" datasets, which are foundational for training and evaluating machine learning models, are not objective, naturally given measurements. Instead, they are the result of human and technological arrangements, shaped by specific choices and contexts that are often invisible or unreported. The authors argue that the machine learning community would benefit significantly from openly discussing these underlying choices and recognizing that reference datasets are contingent, not universal. By focusing on the situated and context-dependent nature of ground truths, the reliability of models can be enhanced. This perspective allows for a better-informed understanding of where, when, and how datasets and the models they influence can be most effectively used. The paper advocates for increasing "situated reliability," which involves clearly articulating the limits and strengths of models and their truth claims. Ultimately, greater attention to the construction of ground truths can foster improved transparency, accountability, and interdisciplinary collaboration within the AI field.