New Framework Enhances Autonomous Driving with Open-Vocabulary Perception and Kinematic Planning.

Autonomous driving systems face a challenge in balancing computational efficiency with the ability to generalize to unforeseen situations. Current methods either rely on computationally intensive dense models that struggle with high-level semantics or efficient sparse models limited by predefined object categories. Furthermore, recent Vision-Language-Action models, while offering open-vocabulary understanding, often conflict with the precise, continuous control needed for vehicle dynamics. A new framework called Lagrange has been developed to tackle these issues. It employs Masked Latent Fields and Vision-Language Models to process class-agnostic object proposals into continuous semantic visual tokens. This approach allows for an open-vocabulary understanding of the environment without the computational burden of dense models or the closed-set limitations of sparse ones. Lagrange frames decision-making as an energy minimization problem, ensuring strict adherence to vehicle kinematics and collision avoidance. Evaluations on both standard and challenging long-tail datasets demonstrate its effectiveness in achieving robust, interpretable, and kinematically feasible autonomous navigation in diverse environments.