Sequence models: RNN, LSTM & GRU

Many business problems involve order: daily sales at a Jaipur sweet shop, a customer's sequence of app clicks, the words in a product review. Feed-forward networks treat inputs as an unordered bag. Recurrent neural networks (RNNs) process one step at a time and carry a memory — the hidden state — from step to step.

The recurrent idea

At every time step t, an RNN combines the current input with the previous hidden state:

h_t = tanh(W_x * x_t + W_h * h_(t-1) + b)

The same weights are reused at every step, so the network can handle sequences of any length. The final hidden state (or all of them) summarises the sequence for a prediction.

The vanishing gradient problem

Training uses backpropagation through time: gradients flow backwards across every step. Multiplying by the same weights again and again makes gradients shrink towards zero (or explode). A plain RNN therefore struggles to remember anything more than 10–20 steps back — it forgets that Diwali sales spiked last year.

LSTM and GRU

The LSTM (Long Short-Term Memory) adds a separate cell state, a conveyor belt of memory, controlled by three gates:

Gate	Question it answers
Forget gate	What old information should be erased?
Input gate	What new information should be stored?
Output gate	What part of memory should be exposed now?

Because the cell state is updated mostly by addition rather than repeated multiplication, gradients survive over long spans.

The GRU (Gated Recurrent Unit) merges gates into two (update and reset) and drops the separate cell state. It has fewer parameters, trains faster and often performs similarly. Try GRU first when data is limited.

Shapes in PyTorch

With batch_first=True, inputs are (batch, seq_len, features).

import torch
import torch.nn as nn

lstm = nn.LSTM(input_size=4, hidden_size=32, num_layers=2, batch_first=True, dropout=0.2)
x = torch.randn(16, 30, 4)          # 16 series, 30 days, 4 features per day
out, (h, c) = lstm(x)
print(out.shape)   # torch.Size([16, 30, 32])  hidden state at every step (top layer)
print(h.shape)     # torch.Size([2, 16, 32])   final hidden state for each layer

Forecasting daily demand

Suppose you have two years of daily units sold for a product, with features for day of week and festival flags. We use the previous 28 days to predict the next day.

import numpy as np

def make_windows(series: np.ndarray, window: int = 28):
    # series: (T, F) array, column 0 is units_sold (already scaled)
    xs, ys = [], []
    for i in range(len(series) - window):
        xs.append(series[i:i + window])
        ys.append(series[i + window, 0])
    return (torch.tensor(np.array(xs), dtype=torch.float32),
            torch.tensor(np.array(ys), dtype=torch.float32))

class DemandLSTM(nn.Module):
    def __init__(self, n_features: int, hidden: int = 64):
        super().__init__()
        self.lstm = nn.LSTM(n_features, hidden, num_layers=2, batch_first=True, dropout=0.2)
        self.head = nn.Linear(hidden, 1)

    def forward(self, x):
        out, _ = self.lstm(x)
        return self.head(out[:, -1, :]).squeeze(-1)   # use last time step

model = DemandLSTM(n_features=3)
X, y = make_windows(np.random.rand(730, 3).astype("float32"))
print(model(X[:8]).shape)   # torch.Size([8])

Important practicalities:

• Scale inputs (e.g. min-max or standard scaling) using statistics from the training period only.
• Split by time: train on the first 20 months, validate on the next 2, test on the last 2. Never shuffle across the split boundary.
• Compare against a naive baseline such as "same as the same weekday last week". If the LSTM cannot beat it, do not ship it.

Text with RNNs

For text, tokens are first mapped to vectors with nn.Embedding, then fed to the RNN:

class ReviewClassifier(nn.Module):
    def __init__(self, vocab_size: int, n_classes: int = 3):
        super().__init__()
        self.emb = nn.Embedding(vocab_size, 100, padding_idx=0)
        self.gru = nn.GRU(100, 128, batch_first=True, bidirectional=True)
        self.fc = nn.Linear(2 * 128, n_classes)

    def forward(self, token_ids):
        _, h = self.gru(self.emb(token_ids))      # h: (2, batch, 128)
        both = torch.cat([h[0], h[1]], dim=1)     # forward + backward states
        return self.fc(both)

A bidirectional RNN reads the sentence both ways, useful for classification where the whole text is available.

Where RNNs stand today

Transformers (next lesson) have replaced RNNs for most language tasks because they process all positions in parallel and handle long-range context better. RNNs remain useful for small streaming or on-device problems, and for modest time-series datasets. For tabular forecasting, gradient-boosted trees with lag features are also a strong, simpler competitor.

Common mistakes

• Random train/test splits on time series, which leak the future.
• Forgetting batch_first=True and feeding (batch, seq, feature) into a layer that expects (seq, batch, feature).
• Scaling with statistics computed on the full dataset.
• Skipping the naive baseline.