New Benchmark Assesses LLM Agent Telemetry for Failure Diagnosis

As AI agent systems become more complex, they increasingly expose execution traces, or telemetry, to help diagnose issues. However, this research questions whether such telemetry is truly sufficient for pinpointing the *origin* of a failure, rather than just detecting its occurrence. The paper introduces TelemetrySuffBench, a controlled benchmark specifically designed to differentiate between failure detection, fault-origin localization, and the ability to safely abstain when evidence is insufficient. TelemetrySuffBench constructs canonical multi-component traces with delayed-binding faults, presenting them in various views, including coarse summaries, seven-factor telemetry masks, and ambiguous origin pairs. Five frontier language models were evaluated using unified protocols. The findings reveal a robust "detection-localization gap": while full telemetry allows for high failure detection, accurately localizing the origin step can be significantly lower (ranging from 33.8% to 97.2% across models). Standard telemetry views (metadata, OpenTelemetry, OpenInference compatible) retained high detection F1 but limited origin-step accuracy to at most 0.5%. Further analysis showed that removing decision content from telemetry drastically reduced origin-step accuracy to zero for all models, and provenance removal also caused large, model-dependent losses. On ambiguous inputs requiring abstention, only three models showed improved abstention with evidence gating, while two models still attempted to answer every case, indicating strong model dependence in safe abstention. The central pattern was reproduced on a frozen holdout set. These results emphasize that while terminal status helps detect failures, reliable causal attribution requires explicit links between decisions and their provenance, along with effective abstention safeguards across different models.