HybridCodec Improves Speech LLMs with Discrete and Continuous Audio

Discrete audio representations have become popular for integrating audio capabilities into Large Language Models (LLMs) and building multimodal text-audio systems. However, a significant drawback is the performance degradation observed in various downstream tasks due to the inherent information loss during the discretization process. To overcome this, researchers introduce HybridCodec, a new framework that models both discrete and continuous representations of speech. It combines temporally compressed discrete tokens with dimensionality-reduced continuous residuals. The core of this approach is a hybridized discrete-continuous focal modulation codec and a hybrid Transformer architecture. This architecture performs autoregressive inference in the discrete domain, complemented by non-autoregressive prediction and continuous residual upsampling. Experimental results demonstrate that HybridCodec substantially improves the retention of speaker characteristics compared to discrete-only methods, while also efficiently reducing the number of necessary autoregressive steps, leading to more efficient and higher-fidelity speech language models.