A model with R-squared near 0 can still give valid 90% prediction intervals - here's why (and the catch)

I recently calibrated a recovery-rate model that had only two weak features. Its point accuracy was almost nothing — R² basically zero. I expected its uncertainty estimates to be junk too. They weren't: the 90% conformal prediction intervals covered ~89% of held-out outcomes. Valid, just wide.

That surprised me enough to nail it down, because it contradicts a belief a lot of us carry around: "my model isn't accurate, so I can't trust its uncertainty." For split conformal prediction, that's backwards. Here's the precise statement, a runnable demo, and the one caveat that actually bites.

Coverage is a property of the procedure, not the model

Split conformal prediction gives a distribution-free, finite-sample marginal coverage guarantee:

P( Y ∈ Ĉ(X) ) ≥ 1 − α