What Are the Basics of Deep Learning You Must Know?

# A simple neural network in Keras

from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense

model = Sequential([
    Dense(64, activation='relu', input_shape=(10,)),   # Hidden layer 1
    Dense(32, activation='relu'),                       # Hidden layer 2
    Dense(1, activation='sigmoid')                       # Output layer
])

model.compile(
    optimizer='adam',
    loss='binary_crossentropy',
    metrics=['accuracy']
)

# model.fit(X_train, y_train, epochs=20)

# This is a 3-layer neural network:
#   - 10 input features
#   - 64 neurons in hidden layer 1
#   - 32 neurons in hidden layer 2
#   - 1 output (binary classification)

# That's it — a full deep learning model in 10 lines.

Line-by-line explanation:

Dense(64, activation='relu', input_shape=(10,))
  → 64 neurons in the first hidden layer
  → Each neuron takes 10 inputs
  → Uses ReLU activation
  → "Dense" = fully connected (each input to each neuron)

Dense(32, activation='relu')
  → 32 neurons in the second hidden layer
  → Each takes 64 inputs (from previous layer)
  → ReLU activation again

Dense(1, activation='sigmoid')
  → 1 output neuron (binary classification)
  → Sigmoid gives a probability (0-1)

optimizer='adam'
  → Algorithm that updates weights to reduce loss

loss='binary_crossentropy'
  → Loss function for binary classification

epochs=20
  → 20 full passes over the training data

Total parameters:
  - Hidden layer 1: 10 × 64 + 64 bias = 704
  - Hidden layer 2: 64 × 32 + 32 bias = 2080
  - Output layer: 32 × 1 + 1 bias = 33
  - Total = 2,817 parameters

Training Loop — simplified:

for epoch in range(20):
    for batch in training_data:
        # 1. Forward pass
        predictions = model(batch.inputs)

        # 2. Compute loss
        loss = loss_fn(predictions, batch.labels)

        # 3. Backward pass
        gradients = compute_gradients(loss, model.weights)

        # 4. Update weights
        for w, g in zip(model.weights, gradients):
            w -= learning_rate * g

# After training:
#   - Model can predict on new data
#   - Weights have been "learned" from data
#   - No human wrote the weights — they emerged from the loop

Key terms to know:

Loss function:
  - Measures how wrong the prediction is
  - Examples: MSE (regression), cross-entropy (classification)
  - The "score of wrongness"

Gradient:
  - The direction and amount each weight should change
  - Computed via backpropagation

Learning rate:
  - How big each weight update is
  - Too high → unstable; too low → slow

Epoch:
  - One full pass over all training data
  - 20-100 epochs is common

Batch size:
  - Number of samples processed before each weight update
  - 32, 64, 128 are typical

Overfitting:
  - Model memorizes training data but fails on new data
  - Solved with more data, dropout, regularization

Underfitting:
  - Model doesn't learn enough
  - Solved with bigger model, more epochs, better features

# CNN example (image classification)

from tensorflow.keras import Sequential
from tensorflow.keras.layers import (
    Conv2D, MaxPooling2D, Flatten, Dense
)

model = Sequential([
    Conv2D(32, (3, 3), activation='relu',
           input_shape=(224, 224, 3)),     # 32 filters, 3x3
    MaxPooling2D(2, 2),                     # downsample
    Conv2D(64, (3, 3), activation='relu'),
    MaxPooling2D(2, 2),
    Conv2D(128, (3, 3), activation='relu'),
    Flatten(),                              # flatten to 1D
    Dense(128, activation='relu'),          # classifier head
    Dense(10, activation='softmax')         # 10 classes
])

# Why CNN works for images:
#   - Conv layers detect edges, shapes, textures
#   - Pooling downsamples (fewer params, translation invariant)
#   - Deep layers detect complex patterns
#   - Reused across image regions (weight sharing)

# This architecture is behind most modern image AI.

Transformers — concept (used in ChatGPT, BERT):

Core idea:
  - Attention mechanism — every word "looks at" every other word
  - No recurrence — parallel processing (much faster)
  - Handles long context better than RNNs

Key components:
  1. Tokenization — text → numbers
  2. Embeddings — numbers → vectors
  3. Multi-head self-attention — context
  4. Feedforward layers — transforms
  5. Stack of N blocks — depth
  6. Output head — classification or generation

Why they won:
  - Parallel (fast on GPUs)
  - Better long-range dependencies
  - Scales with data + compute
  - Transformer decoder → GPT, LLaMA, Gemini

Beginner path to Transformers:
  1. Master feedforward networks
  2. Learn attention conceptually
  3. Try a pretrained model (BERT, GPT)
  4. Only then consider training from scratch

Don't start by training a Transformer — use one.