What we shipped on 2026-07-03
We spent today fighting a ghost in our GPU orchestration, starting with fix(llm): stop setting litellm.api_base global (PR #2082). We had implemented per-model api_base overrides to route vision tasks to a dedicated rail, but requests were still hitting the default port and cold-loading qwen3-vl onto our RTX 5090. The culprit was LiteLLM 1.89.2; its internal logic allowed the module global litellm.api_base to win over per-call kwargs. We had to strip the global assignment entirely to let the overrides actually function.
This was the final piece of a frustrating puzzle involving our second Ollama instance pinned to GPU 1 (PR #2075). The goal was simple: prevent the gemma writer from evicting qwen3-vl and causing qa.vision timeouts. But the path to "pinned" was an obstacle course. First, we found that scheduled-task processes weren't inheriting HKCU user environment variables, meaning our models directory was 404ing (PR #2076). Then, numeric indices for CUDA_VISIBLE_DEVICES proved unreliable on Windows; we had to move to GPU UUIDs to ensure the model landed on the 3090 (PR #2077). Even then, Ollama 0.31's default Vulkan backend was ignoring CUDA pins and grabbing the 5090 anyway. We finally locked it down by setting OLLAMA_VULKAN=false (PR #2078).
