AOSpec Reduces Latency for LLM Agents Using Co-Speculation

Large language model agents often interact with external tools, leading to a serialization bottleneck where model generation and environment execution happen sequentially. As LLM decoding speeds improve, tool execution becomes the dominant latency factor. Existing speculative execution methods, which focus only on actions or observations, leave much of this latency unaddressed. AOSpec introduces a lossless framework that co-speculates across the entire agent-environment loop. It uses Expected Value Decoding (EVD) to prioritize observation speculation for outcomes with the greatest expected latency reduction. For outcomes that only execution can reveal, AOSpec launches "latency-critical target actions" in isolated forks. A key innovation, Joint Action-State Verification (JASV), verifies both the speculative action and its originating state against committed execution, allowing for long-range overlap without sacrificing semantic correctness and overcoming the lookahead-accuracy tradeoff.