New Framework Traces LLM Code Reasoning Lifecycle

Standard accuracy metrics often fail to explain the nuanced behaviors of large language models (LLMs) when performing code reasoning tasks, such as why they might handle variable tracking but struggle with semantically equivalent loops. To gain deeper insight, researchers have introduced a new diagnostic framework that traces the internal lifecycle of code reasoning within LLMs. This framework identifies that models first "brew" an answer, making it linearly recoverable across multiple layers before it becomes self-decodable. Following this brewing phase, the models diverge into one of four distinct resolution outcomes: Resolved, Overprocessed, Misresolved, or Unresolved. Understanding this internal lifecycle is crucial because similar task accuracies can mask fundamentally different failure modes that are not detectable through surface-level evaluation alone. The framework employs layer-wise linear probing combined with Context-Stripped Decoding (CSD). Applied across six code-reasoning task families and 16 models (including Qwen, Llama, and DeepSeek architectures), the study found that all four outcomes carry substantial weight, with overall "Resolved" outcomes at only 41.5%. Controlled experiments varying structure, depth, and operators exposed task-specific failure bottlenecks, such as a significant drop in "Function Call Resolved" with increased call depth. The research indicates that the "brewing scaffold" is a stable empirical regularity across tested decoder-only Transformer families, while resolution success varies with model capability, scale, and training.