Neurosymbolic AI Learns Assembly from Language and Demos

This research introduces a neurosymbolic AI architecture designed to learn the assembly of novel structures, even when encountering unfamiliar parts and semantic constraints not present during initial training. The system focuses on scenarios where an agent, post-deployment, must acquire and utilize new knowledge about valid part types and features through user interactions while attempting assembly tasks. The architecture integrates evidence from both embodied conversations and task demonstrations. This allows the agent to learn from symbolic information encoded in natural language, such as "dump trucks have a dumper," alongside dense visual observations. The study was conducted within a simulated toy truck assembly environment to test its capabilities. Experiments revealed that communicating semantic constraints directly through natural language significantly improved the data efficiency of online adaptation. This approach proved more effective than relying solely on task demonstrations or merely naming parts through natural language, highlighting the power of combining symbolic language understanding with visual learning for complex assembly tasks.