New Metrics Proposed for Reliable and Secure AI Code Generation

This research critically examines the common evaluation metric pass@k used for AI coding agents, identifying a significant misapplication in current benchmarks. The authors argue that pass@k is often incorrectly calculated by conflating the number of unit tests with the number of independent rollout attempts, leading to inflated performance scores. To address this, they propose reliability@k, a corrected estimator that accurately reflects the success rate of independent code generation attempts. Furthermore, the paper introduces security-adjusted reliability@k, a novel metric designed to evaluate not only the functional correctness but also the security safety of generated code. This adjustment counts only those code rollouts that are both functionally sound and free from high-severity insecure patterns. While initial tests with this security lens did not alter agent rankings, the authors present it as a crucial complementary measure for future, more powerful evaluations, emphasizing that functional correctness alone does not guarantee secure code.