(Image credit: Tom's Hardware)
Our local AI testing in 2026 has focused on large language models that can fit entirely into the 128GB of unified memory on Nvidia GB10 and AMD Strix Halo systems. Useful as those smaller open models can be, there is sometimes no replacement for displacement. Today, we’re exploring what’s possible from a local AI cluster with a pair of Nvidia GB10 systems, namely Dell’s Pro Max with GB10 (henceforth Pro Max), which gives us 256GB of RAM for a local AI sandbox.Quantizing an AI model from higher-precision to lower-precision data types involves tradeoffs for performance and accuracy. And even in quantized form, some advanced open models are still too large to fit within 128GB. But those models can be distributed across multiple local systems using the network as a scale-out backbone, just as they are in the data center.Why scale out GB10 systems (or Strix Halos, or Macs)? Local token factories with large VRAM pools built up from discrete GPUs can get crazy, fast. Scaling one to even 128GB of VRAM requires a costly host system with enough PCI Express slots and bandwidth to feed those cards, and going beyond 128GB means spending $20K or more in Nvidia GPUs at a minimum, even if you're building up from older 48GB Ada cards.The preferred recipe for this kind of setup typically includes a Threadripper Pro or Epyc platform, which means a costly CPU, motherboard, and DDR5 kit even before you start adding graphics cards. The power requirements for such a system can quickly get beyond the capabilities of a standard USA 15A circuit (1,800W maximum).And having four discrete GPUs running their blower fans at high speeds under load, along with whatever other active cooling you might need for what is essentially a GPU server, is not going to make for the most pleasant company if you’re sharing a space with it.While a GPU server with four RTX Pro 5000 or RTX Pro 6000 cards is useful for getting the absolute best performance for a given application, those potentially high costs, platform challenges, and quality of life concerns have led local AI enthusiasts to explore other ways of achieving large local memory pools with acceptable LLM inference performance, like the GB10 cluster we’re building today.
