Classic Machine Learning Through the Eyes of an SRE — Part 4
When a computation is too hard, don't compute harder. Change coordinates until it becomes easy.
Every engineer has made this move. Pick the right data structure and the impossible query goes O(1). Re-index the table and the report that took an hour takes a second. Move the problem into a space where it's trivial, solve it there, come back.
That's the kernel trick. SVM's famous move isn't building a curvy model — it's finding a FLAT cut in a transformed space, which corresponds to a curved boundary back in your original features. The separator stays linear in the transformed space. The space did the work.
And here's the part that makes it a trick rather than just a projection: the data never actually goes up there. The optimization only ever needs inner products between pairs of points, and a kernel function computes what that inner product would be in the high-dimensional space, directly from the original coordinates. You get the geometry of a space you never built. Some kernels correspond to infinitely many dimensions, which would otherwise be an awkward amount of memory to allocate.
