You can build the most elegant neural network architecture in the world, and it will learn absolutely nothing until you answer one question first: wrong compared to what, exactly?
That's what a loss function is. It's not a technical afterthought bolted onto the end of a network — it's the thing that turns "prediction" into "learning." Everything from here on, every gradient that flows backward through every layer, starts as a number produced by this one function. Get it wrong, and the rest of the machinery doesn't matter.
So let's actually derive where these loss functions come from, instead of just memorizing "use MSE for regression, cross-entropy for classification" as a rule of thumb.
Loss functions aren't arbitrary — they're probability in disguise
Here's the thing most intro explanations skip: loss functions aren't hand-picked because they "feel right." They fall directly out of maximum likelihood estimation — you assume a probability distribution over your labels, and the loss function is just the negative log of that distribution's likelihood.
