Fast token generation emerges as the key differentiator as heterogeneous inference takes hold
The race for fast token generation has moved from benchmark sheets into production data centers, and the hardware blueprint for winning it is no longer a GPU-only story.
As agentic AI use cases multiply and users demand real-time interactivity, inference infrastructure is being redesigned from the rack up. The divide between compute-heavy prefill and latency-sensitive decode is forcing a new class of purpose-built accelerators into the picture, according to Sid Sheth (pictured, right), co-founder, president and chief executive officer of d-Matrix Corp.
“It is the first public announcement of a heterogeneous compute solution targeted specifically for fast token generation in partnership with NVIDIA,” Sheth said. “We have a lot of inference clouds coming to us and saying — more low latency, deploying that in a GPUs-only infrastructure just doesn’t get us there.”
Sheth and Sudeep Bhoja (left), co-founder and chief technology officer of d-Matrix Corp., spoke with theCUBE’s John Furrier at RAISE Summit 2026, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed d-Matrix’s Corsair production launch, the Parasail-NVIDIA partnership, and the company’s next-generation 3D memory architecture. (* Disclosure below.)
