768GB of cheap Intel Optane DIMM memory sticks used to run 1-trillion-parameter LLM on a system with a single GPU — local Kimi K2.5 install achieved roughly 4 tokens per second

(Image credit: Lenovo)

A Redditor has caused a stir by coaxing a workstation build using Optane PMem DIMMs as RAM to run a 1-trillion-parameter LLM. APFrisco explains in a mini tutorial/guide on the Local LLaMA subreddit how they bought some used Intel Optane Persistent Memory, acquired relatively cheaply second-hand, to “run a 1 trillion parameter model (in this case Kimi K2.5) locally at ~4 tokens/second” on their Xeon workstation.Computer build using Intel Optane Persistent Memory - Can run 1 trillion parameter model at over 4 tokens/sec from r/LocalLLaMACentral to the headlining feat was the Redditor’s sourcing of six Optane PMem (DCPMM) sticks. The discontinued memory format was designed to bridge the DRAM-SSD divide. While the 768GB of Optane (6x 128GB) does indeed offer far lower latency than the best NVMe SSDs, it is still two or three times slower than DRAM. These characteristics are still rather sweet for LLM inference frameworks, and the second-hand price was “much less than what the equivalent DRAM capacity would cost.” But, alas, Optane is dead, so this is an exotic solution.APFrisco’s hardware specs were given as follows:Intel Xeon Gold 6246 CPUTyan S5630GMRE-CGN motherboardAsus Dual GeForce RTX 3060 OC 12GB GPU6x 32GB Samsung 2666MHz DDR4 ECC DRAM sticks6x 128GB Intel Optane DCPMM PC4-2666 NMA1XBD128GQS persistent memory modulesWestern Digital WD SN850X 2TB M.2 2280 NVMe SSDASRock Steel Legend SL-850G 850W 80 PLUS GOLD & Cybenetics Platinum Fully Modular Power SupplySilverstone SST-GD08B (Black) Grandia Series Home Theater PC CaseThe build was configured with the Optane in memory mode and the Samsung DDR4 as cache.The software side of the equation relied on the aforementioned Kimi K2.5’s mixture-of-experts architecture. APFrisco used a hybrid GPU/CPU inference methodology with llama.cpp. Also, to optimize processing, the routing components were shoehorned into the 12GB GPU using llama.cpp’s 'override-tensor' flag.The Redditor is rather proud of the resulting ~4 tokens per second performance. “Given the fact that this is a trillion-parameter frontier-class model running on such a limited hardware budget, I would consider it to be a great success,” writes APFrisco. They go on to lament Intel’s withdrawal of Optane products.If you are interested in this rig rundown and what it achieved in terms of local LLM inference, you can find some more details about the configuration in the source post. Furthermore, APFrisco sticks around in the comments to answer questions. They also appear to benefit from recommendations about how to achieve even better results, given the foundation they have laid.Get Tom's Hardware's best news and in-depth reviews, straight to your inbox.The bigger picture, though, seems to be that there is room for a memory product in the chasm between DRAM and SSDs, particularly for LLMs. Many expect that the gap will soon be bridged by the CXL (Compute Express Link) standard, which promises huge pools of affordable, byte‑addressable memory for these kinds of workloads.