New Fast-Slow AI Architecture Boosts Closed-Loop Driving Performance

A novel asynchronous fast-slow architecture has been introduced to address the latency challenges of integrating large language models (LLMs) into end-to-end driving systems. Traditional methods often compromise by replaying old commands, leading to control outputs that ignore recent observations. This new system overcomes this by employing a frozen 7B vision-language backbone as a "slow" component, which processes navigation instructions and visual history at a lower frequency (5 Hz). It maintains a per-layer key-value cache representing the scene. A "fast" lightweight action expert then leverages this cache and the current camera frame to generate fresh waypoints at a higher frequency (20 Hz, or every 50 ms). To ensure robustness during deployment, the expert is trained with randomized staleness, aligning its training with asynchronous execution. This approach dramatically improves performance on CARLA's LangAuto-Short routes, boosting route completion from 37.0% to 94.0% compared to a frame-skipping baseline. The system also significantly reduces red-light violations and demonstrates strong zero-shot transfer capabilities to unseen towns. The action expert achieves nearly a four-fold reduction in open-loop waypoint error with a low per-tick model cost, making it efficient for real-time vehicle control on consumer GPUs.