Latent Bridge Improves Real-Time AI Game Agents

Developing real-time agents for general computer use, particularly in demanding environments like games, requires balancing rapid action (milliseconds) with long-term planning (seconds). This presents a challenge due to the inherent latency-quality tradeoff: reasoning Vision-Language Models (VLMs) offer effective deliberation but are too slow for real-time control, while reactive VLMs are fast but lack planning capabilities. This study proposes the "Latent Bridge," a novel approach to couple two frozen VLMs of matched scale—a slow reasoning model and a fast reactive model. Unlike a standard "Text Bridge" which relies on text-based communication, the Latent Bridge is a learned continuous channel that projects the slow model's residuals directly into the fast model's input-embedding space, bypassing text round-trips. Evaluations across seven Atari games and a driving domain (MetaDrive) demonstrated that the Latent Bridge consistently matched or surpassed the Text Bridge. It significantly improved performance in games requiring more planning, such as MsPacman (+57%) and RoadRunner (+28%), proving to be a safe and effective drop-in replacement. The benefits are predictable: the bridge helps precisely when slow reasoning already outperforms fast reaction, with gains over a fast-only approach moving together at a high correlation. This research offers a crucial advancement for building more capable and responsive real-time AI agents.