Agentic RAG-VLM Enhances Robotic Grasping with Self-Reflection.

Deploying robotic manipulators in complex, unstructured human environments requires generalizable grasping capabilities, especially in cluttered settings. Existing Vision-Language Model (VLM)-based methods often rely solely on visual similarity for object matching, neglecting crucial physical affordances like handle graspability or material fragility. Furthermore, these systems typically operate open-loop, lacking spatial reasoning and failure recovery mechanisms, which limits their effectiveness with densely packed or physically diverse objects. This research introduces Agentic RAG-VLM, a comprehensive framework designed to bridge the gap between VLM semantic understanding and physically grounded grasp execution. Agentic RAG-VLM integrates retrieval-augmented generation (RAG) with VLMs and agentic self-reflective planning through three interconnected components. First, a Hierarchical Affordance-Aware RAG (HAA-RAG) encodes four-dimensional affordance descriptors (type, material, fragility, graspable region) to retrieve strategies based on functional compatibility rather than just visual appearance. Second, a Scene Graph Constraint Reasoner builds spatial relationship graphs from VLM perception, translating proximity, occlusion, and support constraints into precise grasp parameter adjustments. Finally, an Agentic Self-Reflective Pipeline, equipped with a 14-type failure taxonomy and three-level adaptive retry, enables closed-loop grasp refinement. Evaluated on a 12-task benchmark with 360 trials per configuration, Agentic RAG-VLM achieved an impressive 78.3% overall success rate, representing a 53.3 percentage-point absolute gain over VLM-only baselines, demonstrating the critical importance of affordance-aware retrieval, scene graph reasoning, and agentic recovery for robust manipulation.