Sharding Improves LLM Oversight, Prevents Adversarial Exploitation

Large Language Models, when tasked with evaluating multiple criteria simultaneously, often exhibit a decline in accuracy and reliability. This phenomenon, where an LLM judge's decisions become weakly grounded in evidence as the number of verdicts per call increases, has been observed across various domains like research replication, legal work, and clinical trial assessments. The study found that agreement with human experts decreases as the number of decisions an LLM must make in a single call grows. To address this, researchers propose an intervention called "sharding." This technique involves breaking down the overall evaluation requirements into smaller, manageable groups. Each group is then assigned to a separate LLM call, and the individual verdicts are subsequently aggregated. This approach significantly improves the LLM's agreement with expert judgments, even when compared to a single, more capable LLM receiving the same total computational budget. Furthermore, sharding demonstrates strong resilience against adversarial exploitation. Adversaries can manipulate the presentation of work to increase an overloaded LLM judge's acceptance of unmet criteria. Sharding effectively removes this adversarial advantage, maintaining low error rates even when faced with sophisticated attacks. While it doesn't counter attacks that persuade the judge on each criterion individually, combining sharding with debate-style opposition can address such adaptive re-optimization.