Who&When Pro Benchmarks LLM Failure Attribution in AI Agents.
Summary
This research introduces Who&When Pro, a large-scale benchmark for automated failure attribution in agentic AI systems, featuring 12,326 failed trajectories with golden labels across diverse modalities and scenarios. It reveals systematic patterns in how LLMs attribute failures, offering empirical guidance for future attribution systems.
Why it matters
For professionals developing, deploying, or managing AI agents, accurate failure attribution is critical for debugging, improving system reliability, and accelerating development cycles. This benchmark provides tools and insights to achieve that.
How to implement this in your domain
- 1Utilize the Who&When Pro benchmark to evaluate the failure attribution capabilities of your current LLM-based agent systems.
- 2Analyze the systematic patterns identified in the research to inform the design of your own attribution mechanisms.
- 3Integrate automated failure attribution tools into your AI agent development and monitoring pipelines.
- 4Experiment with different LLM families and protocols for attribution to find the most effective approach for your specific agents.
- 5Contribute to the development of more robust failure attribution systems by leveraging insights from this benchmark.
Who benefits
Key takeaways
- Who&When Pro is a new, large-scale benchmark for automated failure attribution in AI agents.
- It provides 12,326 labeled failed trajectories across diverse scenarios.
- The research reveals systematic patterns in LLM failure attribution.
- This benchmark offers empirical guidance for developing better attribution systems.
Original post by Jiale Liu, Huajun Xi, Shaokun Zhang, Yifan Zeng, Tianwei Yue, Chi Wang, Jian Kang, Qingyun Wu, Huazheng Wang
"arXiv:2607.09996v1 Announce Type: new Abstract: Automated failure attribution uses LLMs to identify where and why agentic systems fail. As agents become more capable, their failures become subtler, making automated attribution increasingly important. We introduce Who&When Pro, a…"
View on XOriginally posted by Jiale Liu, Huajun Xi, Shaokun Zhang, Yifan Zeng, Tianwei Yue, Chi Wang, Jian Kang, Qingyun Wu, Huazheng Wang on X · view source
Want to go deeper?
Turn these trends into skills with Learnijoy's hands-on AI & tech courses.
Explore coursesMore in AI Engineering & DevTools
World Model Depth Benefits Vary in Autoregressive Rollouts
A study on adaptive-compute world models reveals that the benefit of model depth for prediction quality in autoregressive rollouts varies significantly across tasks. It identifies regimes where depth helps, hurts, or has no effect, and shows that training supervision can invert depth's utility.
Model Value Comparisons Skewed by Determinism and Access Clients
Research reveals that comparing values across language models is confounded by response determinism and the specific API or client used to access the model. These factors can significantly alter a model's apparent value profile, making direct comparisons unreliable.
New Framework Analyzes Physics-Informed Neural Networks Training Dynamics
Researchers introduce the Differential Neural Tangent Kernel (DNTK) framework to analyze Physics-Informed Neural Networks (PINNs), establishing its positivity for various network depths and activation functions. This work provides a theoretical foundation for understanding and improving gradient-based training algorithms for PINNs.