Fairness in Generative AI: An Evaluation Problem and Solution

A recent position paper highlights a critical issue in the development of generative AI: the persistent problem of fairness failures, which often reinforce societal inequalities. The authors contend that while multiple factors contribute to these biases, the root cause lies in the current, inconsistent approaches to evaluating fairness. Existing fairness assessments are frequently incomparable across different studies and lack actionable insights for deployment decisions. To address this, the paper advocates for a paradigm shift towards standardized, generative-specific evaluation methodologies. It introduces 'Fairness Cards' as a minimal, structured reporting tool. These cards would explicitly detail evaluation choices, including prompt families, counterfactual protocols, metrics, and refusal handling, thereby fostering greater reproducibility, comparability, and accountability in the assessment of generative models.