LEMUR 2 Benchmark Unlocks Neural Network Diversity for AI Design

Existing Neural Architecture Search (NAS) benchmarks often provide a limited view of the architectural design space, focusing on narrow, task-specific applications and lacking comprehensive cross-domain or deployment-aware evaluations. Addressing these limitations, LEMUR 2 introduces a groundbreaking, large-scale framework designed to foster neural network diversity. This framework integrates generative, evaluative, and deployment pipelines, offering a holistic approach to AI design. LEMUR 2 encompasses an extensive collection of over 14,000 unique neural network architectures and more than 750,000 structured training records. These records detail model performance, hyperparameters, and task outcomes. The diverse architectures were generated using various advanced methods, including AST-based code mutation, genetic and reinforcement learning evolution, fractal architecture generation, and synthesis guided by Large Language Models (LLMs). Beyond generation and training, LEMUR 2 incorporates NN-VR and NN-Lite pipelines for automated deployment and latency benchmarking across heterogeneous mobile and Unity-based VR platforms. This provides crucial real-device performance metadata. The benchmark covers multimodal tasks, image captioning, text-to-image synthesis, and language modeling, enabling in-depth analysis of architectural transferability across different domains. By linking diverse architectures, tasks, and deployment data, LEMUR 2 establishes a new foundation for reproducible, data-driven AI design, significantly advancing the emerging paradigm of LLM-driven AutoML and architectural generalization across modalities and hardware.