LLM Oversight: Shorter Review Units Improve Discrimination

Pre-execution oversight is crucial for ensuring the safety and trustworthiness of AI systems, where an LLM monitor vets actions before they are executed. A key design choice in such systems is the "unit of verification"—how many actions the monitor reviews at once. This research introduces a "twin-prefix framework" to systematically measure the impact of this unit, controlling for other variables. The findings indicate that while longer review windows increase the number of rejections, they do not improve the monitor's ability to discriminate between correct and incorrect actions. Instead, informedness (a measure of discrimination) peaks at one or two actions. This suggests that zero-shot monitors become more rejective, rather than more discriminative, with longer review spans, largely due to observation deprivation. The study advocates for stating the unit of verification in safety cases and co-reporting clean series to avoid misleading conclusions.