ai and ml
NeMo Switchyard brings GPT-5-style model routing to the mainstream
Soaring AI infrastructure costs and model pricing, combined with uncertain returns on investment, threaten to stall enterprise adoption.To make enterprise AI spend a bit more manageable, Nvidia this week unveiled a new software platform that blurs the line between expensive proprietary models and open weights alternatives. Announced alongside Nemotron 3.5-30B-A3B-Lightning, Nvidia’s latest open weights model, NeMo Switchyard is the GPU giant’s latest overture to enterprise. So what exactly is it? Well, it’s a router.
The idea is simple. Switchyard essentially functions as a proxy that sits between the inference server’s API endpoint and the models. But rather than sending every request to the same model, Switchyard can be configured to route prompts to different models in order to optimize for cost, latency, or output quality.
By routing some requests to smaller, cheaper, and potentially locally hosted AI models, Nvidia claims Switchyard can cut job completion costs by 74 percent relative to using Claude Opus 4.8 alone, albeit with an approximately six-point accuracy tradeoff.The right tool for the jobThe key metric in all of this is completion cost rather than price per token. A model might cost one-tenth as much as OpenAI’s or Anthropic’s top model, but if it requires 10x the tokens to complete the request, it isn't actually cheaper.Certain elements of an AI workload may benefit from a larger, smarter model, but not all do. For example, it’d be overkill to ask Claude Opus to generate a title card or summarize a website. It’ll certainly work, but it’ll also cost a fortune compared to Haiku or a locally hosted model that’s been fine tuned just for that purpose. The fewer tokens you burn on the big smart model, the less expensive your API bill is going to be.Nvidia software teams have spent the last several years developing models for this reason. The Lightning model announced this week is only its latest. The 30 billion-parameter MoE model is positioned as a low-latency, general purpose model that can either be used on its own or in conjunction with a larger, smarter model via a router like Switchyard.The company has also developed several application-specific models. Nemotron Parse is one such example. “It’s a small model, one billion parameters, and it’s really good at one task, which is taking a PDF in and then explaining the context inside that PDF whether it’s charts or graphs or tables,” Joey Conway, senior director of AI software and models at Nvidia, explained in a recent interview with The Reg.Many frontier models struggle with this task because PDFs are designed by humans for humans, so by offloading that work to task-specific models, enterprises can not only improve the accuracy of their AI apps, but also reduce costs in the process.
