Learnijoy
NewsAI Engineering & DevToolsAI News & Tools

NVIDIA Compresses Hybrid MoE LLM for Interactive Deployment.

Akhiad Bercovich, Talor Abramovich, Daniel Afrimi, Shay Aharon, Nir Ailon, Vladimir Anisimov, Omer Ullman Argov, Maor Ashkenazi, Tomer Asida, Nave Assaf, Tomer Bar Natan, Alexander Bukharin, Grzegorz Chlebus, Marcin Chochowski, Eric Chung, Mohammad Dabbah, Carlo del Mundo, Ewa Dobrowolska, Ido Galil, Yaniv Galron, Amnon Geifman, Yonatan Geifman, Izik Golan, Alex Gronskiy, Tomasz Grzegorzek, Netanel Haber, Lior Kadoch, Grzegorz Karch, Tomer Keren, Abhinav Khattar, Amir Klein, Tugrul Konuk, Roi Koren, Daniel Korzekwa, Shaun Kotek, Konstantinos Krommydas, Itay Levy, Ofri Masad, Yoav Miron, Pavlo Molchanov, Shahar Mor, Zach Moshe, Saurav Muralidharan, Najeeb Nabwani, Besmira Nushi, Mostofa Patwary, Omri Puny, Johannes Rausch, Tomer Ronen, Sepehr Sameni, Itamar Schen, Elad Segal, Daniel Serebrenik, Ido Shahaf, Soumye Singhal, Daniil Sorokin, Sharath Turuvekere Sreenivas, Marta Stepniewska-Dziubinska, Ali Taghibakhshi, Nima Tajbakhsh, Oren Tropp, Dor Tzur, Anna Warno, Yi-Fu Wu, Michal Zawalski, Jiaqi Zeng, Yian Zhang, Ran Zilberstein, Amit Zuker, Ran El-Yaniv· July 7, 2026 View original

Key takeaways

  • NVIDIA released a compressed Nemotron-3 variant for interactive deployment.
  • Puzzle-75B-A9B achieves 2x higher server throughput and 8x long-context concurrency.
  • Compression uses a multi-stage pipeline including MoE pruning and quantization.
  • The model maintains strong accuracy despite significant size reduction.

Who benefits

Cloud ComputingAI/ML EngineeringSoftware DevelopmentTelecommunicationsGaming

Summary

NVIDIA introduces Nemotron-Labs-3-Puzzle-75B-A9B, a compressed version of Nemotron-3-Super, specifically optimized for interactive deployment. This model achieves significantly higher server throughput and increased long-context concurrency by combining various compression techniques while maintaining strong performance.

NVIDIA has unveiled Nemotron-Labs-3-Puzzle-75B-A9B, a highly compressed variant of their Nemotron-3-Super large language model. This new model is specifically engineered to enhance performance in interactive deployment scenarios, focusing on maximizing server throughput under demanding user constraints. The compression process involved a multi-stage pipeline, integrating techniques such as the Iterative Puzzle compression framework, knowledge distillation, reinforcement learning, quantization, and a Multi-Token Prediction head. These methods jointly optimize heterogeneous Mixture-of-Experts (MoE) pruning, active parameter budget, and Mamba pruning to boost inference efficiency. Performance evaluations show that Puzzle-75B-A9B delivers approximately double the server throughput compared to Nemotron-3-Super on a single 8xB200 node for interactive workloads. Furthermore, for ultra-long-context deployments on a single H100 GPU, it increases 1M-token concurrency from one request to eight. Despite substantial compression, the model retains strong accuracy across a wide range of benchmarks, including reasoning, coding, and multilingual tasks, demonstrating that significant efficiency gains are possible without sacrificing capability.

Why it matters

For professionals deploying large language models, this development offers a path to significantly reduce inference costs and improve responsiveness, making advanced AI capabilities more accessible and scalable for real-time applications.

How to implement this in your domain

  1. 1Evaluate your current LLM deployment infrastructure for potential bottlenecks in throughput and concurrency.
  2. 2Explore compressed LLM variants like Nemotron-Labs-3-Puzzle-75B-A9B for interactive applications.
  3. 3Investigate applying similar multi-stage compression techniques to your own custom models.
  4. 4Benchmark compressed models against their larger counterparts to confirm performance and cost-efficiency gains.

Original post by Akhiad Bercovich, Talor Abramovich, Daniel Afrimi, Shay Aharon, Nir Ailon, Vladimir Anisimov, Omer Ullman Argov, Maor Ashkenazi, Tomer Asida, Nave Assaf, Tomer Bar Natan, Alexander Bukharin, Grzegorz Chlebus, Marcin Chochowski, Eric Chung, Mohammad Dabbah, Carlo del Mundo, Ewa Dobrowolska, Ido Galil, Yaniv Galron, Amnon Geifman, Yonatan Geifman, Izik Golan, Alex Gronskiy, Tomasz Grzegorzek, Netanel Haber, Lior Kadoch, Grzegorz Karch, Tomer Keren, Abhinav Khattar, Amir Klein, Tugrul Konuk, Roi Koren, Daniel Korzekwa, Shaun Kotek, Konstantinos Krommydas, Itay Levy, Ofri Masad, Yoav Miron, Pavlo Molchanov, Shahar Mor, Zach Moshe, Saurav Muralidharan, Najeeb Nabwani, Besmira Nushi, Mostofa Patwary, Omri Puny, Johannes Rausch, Tomer Ronen, Sepehr Sameni, Itamar Schen, Elad Segal, Daniel Serebrenik, Ido Shahaf, Soumye Singhal, Daniil Sorokin, Sharath Turuvekere Sreenivas, Marta Stepniewska-Dziubinska, Ali Taghibakhshi, Nima Tajbakhsh, Oren Tropp, Dor Tzur, Anna Warno, Yi-Fu Wu, Michal Zawalski, Jiaqi Zeng, Yian Zhang, Ran Zilberstein, Amit Zuker, Ran El-Yaniv

"arXiv:2607.04371v1 Announce Type: new Abstract: We present Nemotron-Labs-3-Puzzle-75B-A9B, a compressed variant of Nemotron-3-Super optimized for interactive deployment. We designed the model to maximize server throughput under high user throughput constraints. In interactive ser…"

View on X

Originally posted by Akhiad Bercovich, Talor Abramovich, Daniel Afrimi, Shay Aharon, Nir Ailon, Vladimir Anisimov, Omer Ullman Argov, Maor Ashkenazi, Tomer Asida, Nave Assaf, Tomer Bar Natan, Alexander Bukharin, Grzegorz Chlebus, Marcin Chochowski, Eric Chung, Mohammad Dabbah, Carlo del Mundo, Ewa Dobrowolska, Ido Galil, Yaniv Galron, Amnon Geifman, Yonatan Geifman, Izik Golan, Alex Gronskiy, Tomasz Grzegorzek, Netanel Haber, Lior Kadoch, Grzegorz Karch, Tomer Keren, Abhinav Khattar, Amir Klein, Tugrul Konuk, Roi Koren, Daniel Korzekwa, Shaun Kotek, Konstantinos Krommydas, Itay Levy, Ofri Masad, Yoav Miron, Pavlo Molchanov, Shahar Mor, Zach Moshe, Saurav Muralidharan, Najeeb Nabwani, Besmira Nushi, Mostofa Patwary, Omri Puny, Johannes Rausch, Tomer Ronen, Sepehr Sameni, Itamar Schen, Elad Segal, Daniel Serebrenik, Ido Shahaf, Soumye Singhal, Daniil Sorokin, Sharath Turuvekere Sreenivas, Marta Stepniewska-Dziubinska, Ali Taghibakhshi, Nima Tajbakhsh, Oren Tropp, Dor Tzur, Anna Warno, Yi-Fu Wu, Michal Zawalski, Jiaqi Zeng, Yian Zhang, Ran Zilberstein, Amit Zuker, Ran El-Yaniv on X · view source

Want to go deeper?

Turn these trends into skills with Learnijoy's hands-on AI & tech courses.

Explore courses

More in AI Engineering & DevTools

AI Engineering & DevTools

Zapier vs. Tray: Enterprise Automation Platform Comparison for 2026

This post compares Zapier and Tray.io, evaluating which platform is better suited for enterprise automation needs by balancing power and ease of use. It argues that the best tools scale for complex requirements while remaining intuitive for all users.

Ryan KaneJul 28, 2027
AI ResearchAI Engineering & DevToolsAI News & Tools

Google DeepMind Partners for Breakthrough AI in Gaming

Google DeepMind is collaborating with game studios to develop and prototype advanced AI gameplay, building on 15 years of AI research in the gaming sector, from early titles like Atari to modern complex environments such as EVE Online.

Google DeepMind NewsAug 21, 2026
AI in MarketingAI Engineering & DevTools

AI Visibility Platforms Enhance Marketing Attribution and Workflows

AI visibility platforms offer advanced monitoring capabilities for marketing teams, helping them close citation gaps, link AI search data to CRM attribution, and manage content programs across various regions and workflows.

esantiago@hubspot.com (Erica Santiago)Aug 21, 2026