NVIDIA Accelerates LLM Inference 2.4x Without Full Retraining

NVIDIA has introduced an innovative technique that allows for substantial acceleration of large language models (LLMs) without the need for complete retraining. This method involves creating two copies of an existing model: one 'frozen' copy that handles prompt interpretation and context retention, and another copy specifically trained to generate text in larger chunks rather than word-by-word. By training the second copy with only about 8% of the original data, NVIDIA achieved a 2.4x increase in generation speed while maintaining approximately 99% of the original model's quality.