Process-Aware Fairness Diagnosis for LLM Multi-Agent Systems

As LLM-based multi-agent systems (MAS) are increasingly considered for high-stakes decision-making, such as hiring, relying solely on outcome-based fairness audits can overlook critical points where biases emerge within the decision-making process. To address this, researchers introduce SCOPED-Hiring, a process-aware fairness diagnosis pipeline specifically designed for LLM-based hiring MAS. SCOPED-Hiring operates by constructing controlled resume variants and running them through role-based hiring committees composed of LLM agents. It meticulously logs over 311,000 structured decision trajectories, converting various trajectory fields into quantitative fairness signals. These signals are then organized and analyzed through six diagnostic lenses: final outcome, counterfactual, process, pathway, dynamic, and design effects. The diagnosis revealed that even when final hire rates appear balanced, significant hidden unfairness can exist within the multi-agent decision trajectories. For instance, career gaps often trigger suspicion, proxy cues can unduly influence qualification judgments, and identity cues may lead to unequal investigation processes. Crucially, targeted repairs guided by these process-level diagnoses were shown to reduce the total layered burden of unfairness by 72.3% while only shifting the overall hire rate by a marginal 1.86 percentage points. This demonstrates the power of process diagnosis in guiding effective and precise fairness interventions.