Author(s): The Build Log
Originally published on Towards AI.
Let’s make a case for non-linearity in neural networks, and understand the Universal Approximation Theorem
Stacking a hundred layers in a neural network without non-linear activation functions causes the entire architecture to suffer from linear collapse. Mathematically, every linear layer performs an affine transformation: a combination of matrix multiplication and vector addition, y = Wx + b. Because the composition of any number of affine transformations is itself just another affine transformation, a network with ten, a hundred, or a thousand linear layers simplifies directly into a single matrix operation:
output = Wₑ · x + bₑ
