Activation Functions

An activation function decides whether (and how strongly) a neuron should "fire" based on its weighted input. Without activation functions, stacking layers would be mathematically pointless - a neural network would collapse into a single linear function no matter how many layers it had.

Sigmoid

Squashes any input into a range between 0 and 1, making it useful for binary classification outputs, though it suffers from the vanishing gradient problem in deep networks.

sigmoid(z) = 1 / (1 + e^(-z))

Tanh

Similar to sigmoid but outputs values between -1 and 1, making it zero-centered, which often helps training converge faster than sigmoid.

tanh(z) = (e^z - e^(-z)) / (e^z + e^(-z))

ReLU (Rectified Linear Unit)

The most widely used activation function in modern deep learning - it outputs the input directly if positive, and zero otherwise, making it computationally cheap and effective at avoiding the vanishing gradient problem.

relu(z) = max(0, z)

Softmax

Used in the output layer for multi-class classification - it converts a vector of raw scores into a probability distribution that sums to 1 across all classes.

ReLU is generally the default choice for hidden layers, while Sigmoid or Softmax are typically reserved for the output layer depending on whether the task is binary or multi-class classification.

Coming Up Next