Calibrated Tolerances Boost Tensor Kernel Bug Detection

Ensuring the correctness of tensor kernels, especially in mixed-precision environments, is critical for reliable AI model deployment. Current testing practices often rely on fixed, manually chosen absolute and relative tolerances, which are frequently too loose and fail to catch subtle errors. This study introduces a data-driven approach to calibrate these tolerances. By analyzing the element-wise error distributions from thousands of cloud GPU runs across various tensor kernels and data types, the researchers determined the optimal absolute tolerance justified by correct implementations. This empirical calibration resulted in significantly tighter tolerances, with some cases showing over a 2000x improvement. Applying these calibrated tolerances to LLM-style buggy variants dramatically increased bug detection recall by 9.3 percentage points, from 73.2% to 82.4%, with only a minor increase in false positives. This highlights the importance of operator-aware, data-driven tolerance calibration for robust tensor kernel testing.