SpeechLLMs Gain Word-Level Timestamping Accuracy

This research addresses a gap in Speech Large Language Models (SpeechLLMs), which, despite their advanced speech understanding, often lack fine-grained temporal alignment capabilities. The goal is to transform SpeechLLMs into "temporal-aware content understanding machines" by enabling accurate word-level timestamping. The key innovation involves replacing traditional absolute timestamps with relative time intervals, which results in a more compact vocabulary and improved generalization. To efficiently integrate this capability into pre-trained LLMs, a hybrid fine-tuning strategy is employed: full-parameter fine-tuning for the timestamp-augmented embedding layer and language model head, combined with LoRA fine-tuning for the decoder layers. Furthermore, a novel masked timestamp training objective is introduced. This objective prevents the model from over-relying on perfect ground-truth timestamps, thereby enhancing its robustness when dealing with noisy real-world annotations. Extensive experiments confirm that this approach significantly improves timestamp prediction accuracy while preserving strong speech transcription performance.