Deep networks have a cruel paradox. In theory, more layers should never hurt — the extra ones could just learn to pass their input through unchanged. In practice, before 2015, stacking more plain layers made networks worse: a 56-layer net had higher training error than a 20-layer one. The gradient vanished on its way back to the early layers, and optimisation couldn't even find that "do nothing" identity mapping. ResNet fixed it with almost absurdly little.
The residual reformulation
Instead of asking a block to learn a full mapping H(x), ask it to learn the residual F(x) = H(x) − x, and add the input back:
def forward(self, x):
return F.relu(x + self.f(x)) # y = x + F(x) <- the skip connection
