Modern AI looks like magic from the outside. You type a sentence and a machine writes back something coherent, finishes your function, or turns a paragraph into Japanese. It's tempting to assume something exotic is happening in there.
It isn't. The architecture behind almost every model you've heard of rests on a handful of plain engineering fixes, each one invented to get around a specific, annoying problem. No single genius moment, no secret sauce. Just people noticing their networks were broken and patching them.
This is the story of three of those patches. If you can read a stack trace, you can follow all three.
The wall everyone hit
Around 2014, the recipe for a smarter neural network seemed obvious: make it deeper. More layers meant more capacity, which should have meant better results. Except past a certain point it stopped working. Deeper networks got worse, and not in the way you'd guess.
