LLM Agent Step-Level Credit Signals Fail Causal Audits

This study investigates the efficacy of various step-level credit signals used in training LLM agents, specifically auditing them against causal ground truth derived from executed replays in a single-agent tool environment like ALFWorld. The findings reveal that popular signals, including LLM-judge scores, outcome-conditioned log-probability ratios, and the policy's own confidence, fail to identify causally significant steps at a rate better than random chance. The core issue identified is a divergence between "step correctness" (how existing evaluations grade signals) and "step contribution" (what re-sampling alternatives at each decision point actually changes about the outcome). Causal contribution is found to be sparse, with only about 30.5% of decision points having a measurable effect. Furthermore, the measurability itself is model-dependent. The research concludes that implicit credit often mirrors the policy's fluency, while conditioning on the outcome adds no causal information. Even a confidence-only router, while cutting judge costs, only recovers pivotal steps at chance level. This suggests that current credit assignment mechanisms are not effectively identifying the true causal drivers of agent success.