Nexus Speeds Agentic LLM Tool Routing with KV-Cache Splicing

This research introduces Nexus, a system designed to significantly improve the performance of agentic large language models (LLMs) by addressing the computational burden of tool routing. Traditional methods often re-encode verbose tool schemas with every turn, leading to prefill costs that scale quadratically with sequence length and dominate the time-to-first-token (TTFT). Nexus tackles this by decoupling tool routing from the schema prefill cost. It utilizes an INT8 semantic lookaside buffer (SLB) with a calibrated cross-encoder to select tools via retrieval. Arguments are then generated over a highly compressed textual signature, drastically reducing the need for extensive key/value (KV) cache splicing. This depth-independent approach maintains high routing accuracy even with a large registry of 250 tools, achieving a 1.66x faster first-argument token and saving approximately 80% of main-context tokens. Additionally, Nexus incorporates depth-adaptive KV-cache splicing, where a compiled schema KV block is directly inserted into the live context. While this offers TTFT speedups at moderate depths, it is bounded by rotary position embedding (RoPE) phase drift. Nexus ensures output fidelity by repairing the seam with a depth-adaptive suffix redecode, escalating to a full re-prefill only when necessary, guaranteeing a "never-regress" property on output quality.