Data Scale Drives Cross-Lingual ASR Encoder Transfer, Not Latency

When adapting a streaming Automatic Speech Recognition (ASR) model to a new language, developers often face a choice between initializing the encoder with a multilingual (ML) model or an English-only (EN) model. Conventional wisdom suggests that multilingual encoders offer a greater advantage, especially in low-data scenarios. This study systematically investigates how long this advantage persists, whether tight streaming latency amplifies it, and if it withstands deployment quantization. The researchers conducted a controlled experiment using a 0.6 B-parameter cache-aware FastConformer transducer across eight European languages. They varied target-language data scales from 100 hours to 2500 hours, tested three streaming tiers plus offline decoding, and evaluated up to four public test sets. The key finding reveals that the benefit of multilingual initialization is predominantly tied to the amount of available data, rather than streaming latency. Specifically, the mean word error rate (WER) gap between EN and ML initialization on the FLEURS dataset at 160ms latency decreased from +4.21 percentage points at 100 hours to a mere +0.20 percentage points at 2500 hours. A power-law fit indicates that doubling the target-language data roughly halves the remaining advantage. Across different streaming tiers, the EN-ML gap remained stable at various data scales and approached zero at 2500 hours. Furthermore, 4-bit weight-only encoder quantization, which reduced the encoder footprint by approximately 3x, resulted in only a minor average WER increase of about 0.5 percentage points. The practical guideline derived is to use multilingual initialization for low-data regimes, consider the choice largely irrelevant for large datasets, and make latency and quantization decisions independently.