A Paris-based AI startup just made it significantly cheaper to run large language models. ZML, founded in 2023 by former Zenly VP of Engineering Steeve Morin, has released ZML/LLMD v2, a free, open-source inference server that runs AI models across virtually any hardware platform without rewriting a single line of code.
The software ships as a compact container of roughly 2.4 GB, supports popular model families like Llama 3.1/3.2 and Qwen 3.5, and offers OpenAI-compatible API endpoints. You can swap out your expensive NVIDIA GPUs for AMD chips, Google TPUs, or AWS Trainium accelerators and your AI keeps humming along.
Why a hardware-agnostic inference stack matters
Most production deployments are effectively married to NVIDIA’s CUDA ecosystem. ZML’s approach attacks this problem at its root by using the Zig programming language alongside MLIR and Bazel to compile a single codebase across multiple hardware targets.
The endorsement from Yann LeCun, Meta’s chief AI scientist and a Turing Award winner, adds serious credibility. LeCun praised ZML’s ability to parallelize and run deep learning systems on diverse hardware platforms.
