Counterfactual Fairness Does Not Imply Group Fairness in Images

The relationship between counterfactual fairness (CF) and group fairness (GF) in algorithmic systems has been a subject of active research. While previous studies on tabular datasets suggested that CF might imply GF, this paper investigates this relationship specifically within image classification tasks, where collecting counterfactual samples for sensitive attributes is challenging. To address this, researchers created new image datasets, `oursceleb` and `ourslfw`, by employing high-quality image editing and human annotation. These datasets allow for simultaneous evaluation of both CF and GF. Empirical observations revealed that, unlike tabular data, CF does not imply GF in image classification. The study theoretically attributes this divergence to the existence of latent attributes correlated with, but not caused by, the sensitive attribute. Based on this, a simple baseline called Counterfactual Knowledge Distillation (CKD) is proposed to reduce reliance on these latent correlations. Experiments show that CF-achieving models can satisfy GF if the influence of these latent attributes is successfully reduced using methods like CKD.