LLM-Generated GPU Kernels Often Flawed, New Verifier Reveals.

Current methods for evaluating GPU kernels generated by large language models (LLMs) often rely on loose tests, such as running them on a few random inputs and checking for approximate output similarity. This new research exposes a significant flaw in this approach, demonstrating that many kernels deemed "correct" by these standard tests are, in fact, silently erroneous. A novel "contract-grade" verifier has been developed, incorporating twelve rigorous adversarial gates. These gates check for properties like handling NaNs/infinities, run-to-run consistency, shape invariance, and correct accumulation precision, many of which are tolerance-free. When applied to 2,638 machine-generated kernels previously accepted by a public system's own harness, the verifier found 39.5% to be fundamentally broken and 62.1% to have at least one violation. The findings are robustly defended through multiple independent validations, including positive controls and agreement with reference benchmarks. This work highlights that the perceived correctness rates for LLM-generated kernels are vastly overstated, underscoring the need for more stringent verification tools to ensure reliability in high-performance computing.