The first time I deployed vLLM on OKE, the pod took 12 minutes to become ready. The model download from HuggingFace was 7.5GB. Then the pod crashed (liveness probe, classic mistake), restarted, and downloaded the model again. Another 12 minutes. I burned nearly half an hour watching progress bars.

I added a PVC, which helped — the model persisted across restarts on the same node. But if the pod got rescheduled to a different GPU node? Fresh download. Back to square one.

The fix was obvious in hindsight: pre-stage models in OCI Object Storage and download from there. Same region, private network, 10x faster than HuggingFace.

The Problem With HuggingFace Downloads in Production

HuggingFace is great for browsing and testing. It's not great as a production model source: