Recurrent Neural Networks

A Recurrent Neural Network (RNN) is a type of neural network designed specifically for sequential data - like text, speech, or time series - where the order of inputs matters and past information influences future predictions.

Why Not Use a Standard Neural Network?

A regular feedforward network treats every input independently, with no memory of what came before. This makes it a poor fit for sequences - for example, understanding a sentence requires remembering the words that came earlier in it.

How RNNs Work

An RNN processes a sequence one element at a time, maintaining a hidden state that acts as a memory - carrying forward information from previous steps to influence how the current input is processed.

hidden_state_t = activation(W_input . x_t + W_hidden . hidden_state_(t-1) + b)

Common Applications

Language modeling and text generation
Machine translation
Speech recognition
Time series forecasting, like stock prices or weather patterns

The Vanishing Gradient Problem

Standard RNNs struggle to remember information over long sequences, because gradients tend to shrink to almost nothing as they're propagated back through many time steps - this led to the development of more advanced variants like LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) networks, which are specifically designed to retain long-term dependencies.

Implementing in Python

from tensorflow.keras import layers, models

model = models.Sequential([
    layers.SimpleRNN(64, input_shape=(timesteps, features)),
    layers.Dense(1)
])

Unlike CNNs, which are ideal for spatial data like images, RNNs are built for temporal or sequential data - the two architectures are often combined in tasks like video analysis, where both space and time matter.

Coming Up Next

This wraps up the deep learning architectures covered in this course - from Convolutional Neural Networks and their role in image recognition, through pooling and fully connected layers, the landmark AlexNet case study, to Recurrent Neural Networks for sequential data. Next, you'll dive deeper into Sequence Modelling and the training challenges that come with it.