CoT Training Improves LLM Agent Actions, Not Just Reasoning Faithfulness

Research explores the impact of Chain-of-Thought (CoT) training on large language model (LLM) agents, specifically questioning whether CoT improves the agent's ability to modify its actions through generated reasoning or if it primarily enhances direct action prediction from the prompt. Previous work suggested CoT might sometimes be post-hoc reasoning rather than genuine deliberation. The study compares "prompt actions" (actions predicted without CoT) with "CoT actions" (actions predicted with CoT). Findings indicate that CoT training substantially improves the quality of prompt actions across various checkpoints. Interestingly, the relative performance gap between CoT actions and prompt actions remains largely consistent, suggesting that CoT training doesn't necessarily make the explicit reasoning process more impactful, but rather strengthens the model's underlying ability to determine the correct action. Furthermore, later checkpoints show a reduced tendency to revise actions based on CoT, implying an increased reliance on the initial prompt. An intervention involving selectively masking action-token supervision during training was found to improve out-of-domain generalization, offering a potential method to enhance model robustness.