Weight Calculation

Weights are the core learnable parameters of a neural network - they determine how much influence each input has on a neuron's output, and adjusting them correctly is the entire goal of training.

Weight Initialization

Weights are typically initialized with small random values rather than zeros - if every weight started identical, every neuron in a layer would learn the exact same thing, a problem known as the symmetry problem.

import numpy as np

weights = np.random.randn(num_inputs) * 0.01

How Weights Get Updated

During training, weights are updated in the direction that reduces the model's error, guided by the gradient of the loss function with respect to each weight:

w_new = w_old - learning_rate * gradient

Role of the Learning Rate

The learning rate controls how big each weight update step is. Too high, and training can overshoot and become unstable; too low, and training becomes painfully slow, potentially getting stuck before reaching a good solution.

Weights vs Bias

While weights control the steepness/direction of the decision boundary, the bias term shifts it - allowing the network to fit patterns that don't pass through the origin.

Poor weight initialization is a common cause of neural networks failing to train properly - techniques like Xavier and He initialization were specifically designed to keep signal strength stable across deep layers.

Coming Up Next