Ex-Omni-2D Creates Expressive Omni-Modal Dialogue with Visual Avatars

While omni-modal dialogue models can process multimodal inputs and generate spoken replies, their responses typically lack a visual embodiment. A new framework, Ex-Omni-2D, aims to bridge this gap by generating a fully coordinated response that includes text, personalized speech, and a reference-conditioned video, effectively giving AI avatars a native visual presence. Given a multimodal query, along with a reference image and audio, Ex-Omni-2D first predicts a structured "Visual Thought Plan" (VTP) that outlines scene, emotion, and motion. This is followed by the generation of response text and native multi-codebook speech units. These speech units form a crucial shared acoustic-temporal interface, allowing them to be decoded into speech and simultaneously aligned online with video frames. This innovative interface enables the model to learn both response and avatar pathways from diverse speech, dialogue, and avatar-video datasets, eliminating the need for extensive query-text-speech-video supervision. For efficient incremental generation, the framework distills a full-sequence Video Generator into a few-step block-causal "Streaming Student." This student model uses a Prefix Streaming mechanism to maintain a clean latent representation across consecutive chunks, minimizing cumulative degradation in later parts of the video. The complete four-GPU pipeline achieves an end-to-end real-time factor of 1.293, demonstrating a practical balance of quality and efficiency for generating expressive, visually embodied AI dialogue.