Sequence Modelling

Sequence modelling refers to the broader task of building models that understand, predict, or generate ordered data - where the position and context of each element matters, not just its individual value.

What Makes Sequential Data Different

Unlike tabular data where each row is independent, sequential data - like a sentence, an audio waveform, or a stock price over time - carries meaning through order. Shuffling the words in a sentence, or the days in a stock chart, would destroy the information entirely.

Common Sequence Modelling Tasks

Sequence-to-Value - e.g. sentiment analysis, predicting a single output from a sequence of words
Sequence-to-Sequence - e.g. machine translation, converting one sequence into another
Sequence Generation - e.g. text generation, predicting the next element in a sequence step by step

Architectures Used

RNNs (covered earlier) were the traditional go-to for sequence modelling, along with their more capable variants LSTM and GRU, which handle longer sequences better. More recently, Transformer architectures have largely taken over for tasks like language modelling, since they can process entire sequences in parallel rather than one step at a time.

A Simple Example

from tensorflow.keras import layers, models

model = models.Sequential([
    layers.Embedding(input_dim=5000, output_dim=64),
    layers.LSTM(64),
    layers.Dense(1, activation="sigmoid")
])

Choosing the right sequence length (how far back the model looks) is an important design decision - too short and the model misses important context, too long and training becomes slower and harder to converge.

Coming Up Next