Second Thought Boosts LLM Agent Efficiency with Parallel Reasoning

The paper proposes "Second Thought," an innovative, training-free inference framework designed to enhance the efficiency of Large Language Model (LLM) agents. Traditional ReAct-style agents typically confine deliberate reasoning to a "Thought" phase, leaving their reasoning capabilities idle while they execute actions and await environmental observations. Second Thought addresses this by utilizing these "idle windows." The framework operates by forking four auxiliary reasoning branches immediately after each Thought phase concludes. These branches decode concurrently with the main agent loop. Once the environment observation arrives, the generated thoughts from these parallel branches are merged back into the main process, effectively relocating additional reasoning off the main thread's sequential decoding path. Evaluations across three agentic benchmarks and three reasoning LLMs demonstrated significant improvements. Second Thought consistently lowered the average turn count in all nine model-benchmark pairs and reduced main thread decoding in six of them by up to 43%, averaging around 20% in those settings. Crucially, Pass@1 scores remained largely unchanged or even improved, indicating enhanced efficiency without sacrificing performance. Compared to a compute-matched control, Second Thought achieved higher Pass@1 with less sequential decoding.