Running ML workloads at scale means dealing with GPUs. And GPUs are annoying. They're scarce, they're fragmented across clouds, and every provider has its own way of doing things. H200s here, L4s there, different APIs, different configurations. When you just want to train a model, the last thing you need is to become an expert in three different cloud consoles.
At Shopify, ML work touches almost everything. Training all this machinery needs a lot of GPUs.
We use SkyPilot for all our training workloads. It's an open-source framework that lets you define jobs in YAML and run them on whatever cloud has capacity. You say what you need (GPUs, memory, disk) and the system figures out where to put it.
SkyPilot comes equipped with many wonderful features, but we needed to make it work for an organization like ours. This required some expansions to support multi-team management, cost tracking, fair scheduling, the whole thing.
The architecture
