AI agents have expanded inference from single-turn interactions into multi-step workflows that reason, invoke tools, coordinate subagents, and carry growing context from one turn to the next. The scale of this shift is now visible in raw consumption: across 100 trillion tokens of real-world usage, OpenRouter’s State of AI report found that average prompt tokens per request grew roughly fourfold, and single agentic requests consume 15 times the tokens of ordinary chat.
Properly characterizing hardware performance for this workload presents new challenges. A useful benchmark must capture long-context prefill, KV-cache reuse, interactive decode, tool-call gaps, and distributed mixture of experts (MoE) execution under realistic concurrency. It must also show how much of the AI-factory power budget is converted into useful agentic throughput while maintaining an acceptable user experience.
This post examines SemiAnalysis AgentX, a benchmark that evaluates AI infrastructure for agentic-coding inference using replayed production-style sessions. It highlights Vera Rubin NVL72 preview results showing up to 30x higher AI-factory throughput per megawatt than GB300 NVL72, while showing that Blackwell GB300 NVL72 extends its order-of-magnitude throughput-per-megawatt advantage over prior generations to dynamic agentic workloads.
