Uncertainty Signals Improve LLM Code Generation with Rollback

This research investigates the utility of uncertainty signals in enhancing the performance of large language models (LLMs) for code generation. Specifically, it focuses on an inference-time strategy called uncertainty-aware rollback decoding. This method leverages uncertainty metrics to detect potentially erroneous or unreliable segments within the generated code. Upon identification, the system can "roll back" to a previously valid prefix, allowing for a more robust and accurate generation process without requiring model retraining. The study conducted a comprehensive evaluation across seven code LLMs, five code generation benchmarks, and eight token-level uncertainty signals. The findings demonstrate that the complete rollback framework consistently outperforms an equal-budget restart strategy. Significant improvements were observed, with gains of up to 0.26 in pass@1 and 0.35 in AvgTestPassRate on functional code generation tasks, and an absolute improvement of up to 6.4% in Patch-Aligned Safe Rate on Dsec-Python. Information-theoretic measures like token entropy and negative log-likelihood proved to be the most effective uncertainty signals.