Silent Failures Undermine Multimodal Agentic Search Reliability

Multimodal agentic search systems, which increasingly use external tools to answer complex visual questions, often exhibit hidden reliability issues termed "silent failures." These failures occur when a system produces a seemingly correct final answer, but its internal search trajectory or evidence grounding is fundamentally flawed. This research introduces a six-category taxonomy to classify these silent failures, including issues like modality shortcuts, phantom grounding, and provenance hallucination. To diagnose these problems, the study developed a trajectory-level evaluation pipeline that assesses both answer correctness and the quality of evidence grounding using a unified ReAct-style scaffold. Experiments conducted on MMSearch-Plus trajectories across four leading multimodal models consistently showed that simply evaluating surface accuracy significantly overestimates the true correctness of the underlying reasoning process. Further cross-judge validation, stress tests with blank images, and tool ablations confirmed that these silent failures are capability-dependent and tend to shift rather than disappear entirely, highlighting a persistent challenge in ensuring the robustness of these advanced AI systems.