Originally published on kuryzhev.cloud
Your anomaly model tested perfectly offline — 94% precision on a historical CSV export, clean ROC curve, the works. Then it went live wired to real query_range calls, and within a day it was paging on-call for pod restarts and empty scrape windows. This is one of the most common failure modes we've hit rolling out predictive alerting on top of Prometheus, and it almost never shows up in the training notebook. It shows up in production, at 3am, as noise.
Symptoms
The tell is usually a mismatch between what the model was trained to expect and what it actually receives at inference time. On our setup (Prometheus 2.48.x, Python 3.11, pandas 2.1.x, scikit-learn 1.4.0) we saw four recurring patterns:
Predictive alerts firing right after a Prometheus restart or a scrape gap — the model scores a stale window as a spike.
