Guava Harness Unlocks Embodied AI Capabilities in Smaller Models

Large language models, when combined with extensive vision-language data, show significant promise for developing embodied AI agents. Instead of building complex end-to-end systems, a more effective approach involves using a "harness" that allows these models to utilize external tools for perception, planning, and control, thereby integrating high-level reasoning with specialized modules. This research introduces Guava, a novel harness framework specifically designed for embodied tool use. Through systematic exploration, the team identified three critical components for effective embodied agents: continuous perception-reasoning-action cycles, abstracting actions semantically, and incorporating diverse multimodal observations. These principles are crucial for enabling robust manipulation capabilities. To validate the universality of these design principles, the researchers developed a training pipeline that distills advanced manipulation skills into a compact 4-billion-parameter open-source model. This was achieved using a remarkably small dataset of under 2,000 simulated trajectories. Both simulated and real-world experiments confirmed that this smaller model, when equipped with Guava, could perform comparably to leading proprietary models, demonstrating strong generalization across new objects, instructions, and complex, long-duration tasks. This highlights the potential for scalable, model-agnostic interfaces to empower even smaller models with sophisticated embodied intelligence.