Deep Learning Glossary Key Words Explained Simply

# A single neuron in plain Python
inputs  = [0.5, 0.8, 0.2]
weights = [0.4, 0.1, 0.7]
bias    = 0.1

# Step 1: weighted sum
total = sum(i * w for i, w in zip(inputs, weights)) + bias
# 0.5*0.4 + 0.8*0.1 + 0.2*0.7 + 0.1 = 0.48

# Step 2: activation (simple ReLU)
output = max(0, total)   # 0.48
print(output)

Input Layer → Hidden Layer → Output Layer
    (3)           (4)            (1)

Each arrow = a weight.
The network learns by adjusting these weights
so the final output matches the correct answer.

def relu(x):
    return max(0, x)

print(relu(-3))   # 0
print(relu(0))    # 0
print(relu(5))    # 5

# ReLU is simple, fast, and works well in deep networks.
# It blocks negatives, which helps avoid "vanishing gradients".

import math

def softmax(scores):
    # Subtract max for numerical stability
    m = max(scores)
    exps = [math.exp(s - m) for s in scores]
    total = sum(exps)
    return [e / total for e in exps]

print(softmax([2.0, 1.0, 0.1]))
# [0.659, 0.242, 0.099]  → sums to 1.0

for epoch in range(num_epochs):
    for batch in dataloader:          # split data into batches
        predictions = model(batch.X)  # forward pass
        loss = loss_fn(predictions, batch.y)
        loss.backward()               # compute gradients
        optimizer.step()              # update weights
        optimizer.zero_grad()         # reset gradients

    print(f"Epoch {epoch}, Loss = {loss.item():.4f}")

Suppose you have 1,000 images.

Batch size = 100  →  10 batches per epoch
Epochs = 10       →  10 full passes over the data

Total weight updates = 10 epochs × 10 batches = 100 updates.

Forward pass:
  Input → Hidden → Output → Loss

Backward pass (backprop):
  Loss → Output → Hidden → Input

At each step, we ask:
  "If I nudge this weight slightly,
   how does the loss change?"

That number is the gradient.
We move each weight in the opposite direction
of the gradient — that's gradient descent.

# Stochastic Gradient Descent (plain)
weights -= learning_rate * gradients

# Adam (adaptive)
# Keeps a running average of past gradients
# and adapts the step size per weight.
#
# Faster convergence, less tuning.

optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

# Each word becomes a 3-number vector.
# Words with similar meaning end up close together.

king  = [0.8, 0.5, 0.2]
queen = [0.7, 0.6, 0.2]
apple = [0.1, 0.9, 0.4]

# Distance between king and queen is small.
# Distance between king and apple is large.

# Meaning is captured by position in space.

Famous trick (Word2Vec):
  king - man + woman ≈ queen

The model learns that "king" and "queen"
differ mainly in gender — without being told.
This is the magic of embeddings.

Pretrained model (trained on billions of words)
         ↓
Fine-tuning (your data: 1,000 examples)
         ↓
Model adapted to your task

Why it works:
- The model already "knows" language.
- You just teach it your specific style.

Example: Fine-tune GPT-2 on your company's
support emails to build a custom assistant.

Transformer block:

  Input tokens
      ↓
  Multi-Head Self-Attention
      ↓
  Add & Norm
      ↓
  Feed-Forward Network
      ↓
  Add & Norm
      ↓
  Output tokens

Stack many blocks → GPT / BERT.
The magic: attention lets every token
look at every other token in parallel.