HybridCodec Improves Speech LLMs with Discrete and Continuous Audio Representations.

Large Language Models (LLMs) are increasingly integrating audio capabilities, often relying on discrete audio representations. However, this discretization can lead to information loss and performance degradation in downstream tasks. Researchers have introduced HybridCodec, a novel framework designed to mitigate these issues by combining both discrete tokens and continuous residuals. The HybridCodec architecture features a hybridized discrete-continuous focal modulation codec and a hybrid Transformer. This setup allows for autoregressive inference in the discrete domain while simultaneously performing non-autoregressive prediction and continuous residual upsampling. Experimental results indicate that this hybrid approach significantly enhances the retention of speaker characteristics compared to methods that rely solely on discrete representations. Furthermore, it achieves this improvement while also reducing the number of necessary autoregressive steps, suggesting greater efficiency.