Why Deep Sigmoids Stop Learning

Question

Compute the maximum of sigma'(x) for the sigmoid sigma(x) = frac11+e^-x. Use it to show why stacking sigmoid layers starves early layers of gradient, explain what ReLU changes, and name ReLU's own failure mode.

Accepted Answer

The maximum. sigma'(x) = sigma(x)big(1 - sigma(x)big), and p(1-p) peaks at p = frac12, i.e. x = 0: max sigma' = tfrac12 cdot tfrac12 = mathbf0.25. (Grid check confirms 0.2500.) The starvation argument. Backprop's chain rule threads one sigma'(z_ell) (times a weight) through every layer between the loss and an early parameter. Even in the best case, all pre-activations sitting at zero, the product of L sigmoid derivatives is at most 0.25^L: ten layers Rightarrow le 0.25^10 approx 9.5times10^-7. In the realistic case units saturate (|z| large, sigma' approx 0) and the factors are far smaller. Early layers receive gradients attenuated to numerical dust: the deep net trains its last layers and leaves the first ones at initialization. That is the vanishing-gradient problem, and it capped useful depth for years. What ReLU changes. max(0, x) has derivative exactly 1 on its active half, a gradient passing through active units is unattenuated: products of ones, not products of quarters. Combined with sensible initialization and normalization, this single substitution unlocked depth. (Skip connections attack the same problem from the architecture side: an identity path gives the gradient a h

Why Deep Sigmoids Stop Learning

Problem

Your answer

Hints