Multi-Agent Pipelines Don't Reduce LLM Bias, Audit Capacity Does

This research investigates whether distributing a critical decision, like resource allocation in a disaster triage scenario, across a multi-agent LLM pipeline helps mitigate demographic bias compared to a single LLM agent. Using a synthetic simulator with GPT-4o-mini, the study compared a single-agent control to a nine-agent pipeline under varying pressure conditions. The findings revealed no measurable difference in the frequency of biased outcomes between the two setups. Crucially, the study found that the capacity of the independent audit step was the primary factor in whether bias was detected. When auditors were overloaded, a significantly higher percentage of biased outcomes went undetected, primarily because fewer cases were reviewed at all, rather than a degradation in judgment for reviewed cases. A follow-up experiment showed that prioritizing the audit queue by estimated risk could recover much of the lost coverage under capacity constraints. This highlights that pipeline design alone doesn't eliminate bias; effective audit mechanisms, especially under resource limitations, are paramount.