Exploding Gradient Problem

The exploding gradient problem is the opposite of the vanishing gradient problem - it occurs when gradients grow uncontrollably large as they're propagated backward through a deep network, causing unstable and erratic training.

Why It Happens

Just as repeatedly multiplying small numbers causes gradients to vanish, repeatedly multiplying numbers greater than 1 across many layers causes gradients to grow exponentially - this is especially common in deep networks and RNNs processing long sequences.

Signs of Exploding Gradients

Model weights become extremely large or turn into NaN values
Loss fluctuates wildly or suddenly spikes during training
The model fails to converge even after many epochs

Gradient Clipping

The most common fix is gradient clipping - capping the gradient's value (or norm) at a maximum threshold before it's used to update the weights, preventing any single update from being too drastic.

from tensorflow.keras import optimizers

optimizer = optimizers.Adam(clipnorm=1.0)

Other Solutions

Use a smaller learning rate
Apply proper weight initialization
Use Batch Normalization to stabilize activations across layers
Use LSTM/GRU cells instead of plain RNNs for long sequences

Gradient clipping is especially common when training RNNs on long sequences, where exploding gradients tend to appear far more often than in standard feedforward networks.

Coming Up Next