Oblivious Audits Prevent AI Model Manipulation in Fairness Checks

Algorithmic governance increasingly relies on audits to scrutinize machine learning models, especially for fairness. However, current audit practices are vulnerable to manipulation, as model providers can often detect or infer an audit, allowing them to strategically alter model behavior or equalize fairness metrics to pass inspection. This paper addresses this critical vulnerability. The research proposes a new audit protocol designed to make such manipulations much harder to hide and easier to detect. The core of the approach is an "oblivious" auditing mechanism, which leverages Private Information Retrieval (PIR). This mechanism compels the model provider to label a large dataset without revealing to them which specific subset of instances the auditor will ultimately use for the actual audit. This method is efficient, imposes minimal overhead on the auditor, and crucially, requires no modifications to the audited model, its training, or inference pipeline. Theoretical guarantees show that providers attempting to hide unfairness under this protocol must falsify a significantly larger number of responses, thereby increasing both the difficulty and the likelihood of detection. Experimental results confirm the practicality and effectiveness of this manipulation-proof approach.