We built a feature that dominated our model's importance rankings. Across four random seeds and three quantiles (q10, q50, q90), it ranked first by a wide margin. The gain scores were stable and reproducible. Every diagnostic said it was working.
It made our model measurably worse out of sample.
This is a post-mortem on why tree-based models eagerly prioritize high-variance target encodings that fail to generalize, and how to detect when your feature engineering has collided with an irreducible noise floor.
The Setup
We run a quantile regression model (LightGBM, predicting q10/q50/q90 price ranges) for Flyback.ai, a buy-side intelligence platform for the luxury watch market, where labels are inherently noisy. The same reference, listed twice by two different sellers, will sell for materially different prices. This is not measurement error. It is structural market variance driven by unobserved factors that no feature matrix can fully capture.
