Agentic AI Workflows Demand New Server Architectures.

The emergence of agentic AI in datacenters presents significant architectural challenges that have been largely unexplored. A new study, including production data from Microsoft Azure and controlled experiments with open-source frameworks, characterizes these agentic workflows. It finds that execution is highly fragmented and heterogeneous, involving frequent transitions between LLM inferences, tool invocations, and orchestration decisions that repeatedly cross the CPU-GPU boundary. This fragmentation leads to specific resource demands: the CPU often sits on the critical path due to orchestration and tool execution, while load patterns are bursty with sudden spikes. The composition of models and diversity of tasks further impact GPU utilization. These characteristics expose inefficiencies in uniform server designs, as fragmented execution strands CPU and GPU capacity, and homogeneous CPU provisioning is inefficient for diverse software roles. To address these issues, the researchers propose architectural implications for agentic servers and introduce Agora, a prototype system. Agora dynamically reclaims idle CPU cores for co-located tasks, protects agent tail latency, oversubscribes GPU memory by prefetching, and uses affinity-aware scheduling to improve locality. These innovations significantly enhance utilization and throughput while maintaining performance, pointing towards future server designs optimized for agentic AI.