A code deploy either works or throws. A model deploy usually does neither: the new version returns 200s at the same latency and is slightly worse at the thing you care about. That is why model rollouts need a quality gate as well as a health gate, and why the honest question about a canary is not “did it error?” but “have I seen enough requests to tell?”
Why model deploys are not code deploys
Three properties break the usual assumptions.
Failure is silent and distributional. A 2% drop in extraction accuracy produces no errors, no latency change and no alert. It shows up as support tickets three weeks later.
Output is non-deterministic. You cannot diff two responses and conclude anything from one pair. Comparison has to be statistical, over a sample.
