Vanishing Gradient Problem

The vanishing gradient problem occurs when gradients become extremely small as they're propagated backward through a deep network during training, causing the earlier layers to learn extremely slowly - or not at all.

Why It Happens

During backpropagation, gradients are calculated using the chain rule, which involves multiplying many small numbers together layer by layer. Activation functions like Sigmoid and Tanh squash their outputs into a narrow range, and their derivatives are often less than 1 - when many of these small derivatives get multiplied across many layers, the resulting gradient shrinks toward zero.

gradient_layer1 = derivative_L * derivative_(L-1) * ... * derivative_1

Why It's a Problem

If the gradient reaching the early layers is close to zero, their weights barely get updated - meaning the network effectively stops learning in those layers, even though the later layers may still be training normally.

Common Solutions

• Use ReLU activation instead of Sigmoid/Tanh, since its derivative doesn't shrink for positive inputs
• Use proper weight initialization techniques like Xavier or He initialization
• Use architectures with skip connections, like ResNet, which let gradients flow more directly to earlier layers
• Use LSTM/GRU cells in place of plain RNNs, which are specifically designed to preserve gradient flow over long sequences
• Apply Batch Normalization to keep activations in a stable range throughout training

Coming Up Next

The opposite issue can also occur during training - gradients growing uncontrollably large instead of shrinking - known as the Exploding Gradient Problem.