Atrex-Bench and AKA Evaluate LLM-Generated GPU Kernels

Existing benchmarks for GPU kernel generation often use synthetic problems, which don't accurately reflect real-world production workloads. To provide a more realistic assessment, Atrex-Bench has been developed. This new benchmark comprises 30 operators and 440 shapes directly sampled from full-cluster production inference traces, weighted by their observed GPU time. Evaluations using Atrex-Bench show that even the most advanced vanilla coding agents achieve only about 10% of the hardware's theoretical peak performance on these production operators. Furthermore, apparent correctness often stems from PyTorch fallbacks rather than truly optimized, model-generated kernels. To close this significant performance gap, researchers also released Atrex-Kernel-Agent (AKA). This profile-driven optimization agent employs an iterative measure-revise search process, incorporating "optimization dropout" to escape stalled search contexts. It leverages a layered GPU-optimization knowledge base, including 298 reference kernel files and 244 optimization documents, along with external API/ISA references. A controlled case study demonstrated AKA's ability to convert previously unoptimized fallbacks into high-performing kernels. These kernels either matched or surpassed the performance of hand-tuned production baselines, highlighting the agent's effectiveness in generating production-ready GPU code.