Gradient Descent

Gradient Descent is the optimization algorithm that neural networks use to minimize error by gradually adjusting weights in the direction that reduces the loss the most.

The Core Idea

Imagine standing on a hilly landscape (the error surface) in thick fog, trying to reach the lowest point. You can't see the whole landscape, but you can feel the slope beneath your feet - gradient descent works the same way, taking small steps in the direction of the steepest downward slope.

The Update Rule

w_new = w_old - learning_rate * (dLoss/dw)

The gradient (dLoss/dw) tells you the direction of steepest increase in loss, so subtracting it moves the weight in the direction that decreases the loss.

Types of Gradient Descent

Batch Gradient Descent - uses the entire dataset for each update; accurate but slow
Stochastic Gradient Descent (SGD) - uses one random sample per update; fast but noisy
Mini-Batch Gradient Descent - uses small batches; the most common choice in practice, balancing speed and stability

Implementing in Python

from sklearn.linear_model import SGDClassifier

model = SGDClassifier(loss="log_loss", learning_rate="optimal")
model.fit(X_train, y_train)

Choosing the right learning rate is critical - too large and gradient descent can overshoot the minimum and diverge, too small and training can take an impractically long time to converge.

Coming Up Next