Conformal prediction is the easiest way to put a calibrated uncertainty band around any model: wrap a point predictor, and you get intervals with a finite-sample coverage guarantee — no distributional assumptions. It's deservedly popular.
There's a catch that bites in production: that guarantee is marginal and it assumes exchangeability. The moment your data drifts — almost any time series, any online-serving setting — exchangeability is gone, and split-conformal silently stops delivering the coverage it promises. No error, just a band that's quietly too narrow.
Here's the failure, then a fix that actually holds, with runnable code.
The failure, measured
Target 90% intervals. Residuals whose spread drifts upward over time (a textbook covariate/heteroscedastic shift). Calibrate split-conformal on the first chunk and let it run:
