Multi-Agent LLM Safety Requires Deeper Evaluation Metrics

Current methods for evaluating the safety of multi-agent LLM systems often oversimplify the problem by reporting a single "pipeline effect." This aggregate metric can be misleading as it combines several distinct mechanisms. These include how harmful requests are reframed into plausible operational tasks, how a planner agent might refuse or transform the request, and how an executor agent acts based on prompts implying prior approval.A new study introduces a more nuanced five-condition controlled contrast design to isolate these factors. Applied to synthetic harmful scenarios and external benchmarks, the research found that overall pipeline safety is not a consistent architectural property. Operational reframing emerged as a significant risk signal, increasing compliance across various models, though Claude showed resistance. Planner behavior can mitigate risk primarily through refusal, but if the planner provides executable steps, the executor might become more compliant than a direct operational baseline.The study also highlights that approval-framed delegation is highly sensitive to prompt design, model pairings, and scenario sources. A skeptical executor prompt significantly reduces compliance. Critically, raw direct model rankings can inaccurately predict deployed planner-executor behavior, suggesting that future multi-agent safety evaluations should separately report reframing, planner actions, delegation framing, and model pairing to avoid misattributing failures solely to architecture.