RNN / LSTM: Rise and Fall of Sequence Models

For sequence modeling — text, speech, time series — there was a 25-year era between MLPs and Transformers. RNNs and LSTMs.

Understanding their limits is what makes Transformers’ victory make sense.

The Problem: Sequences

MLPs treat each input independently. But many tasks need memory:

• Translate this sentence (depends on word order)
• Predict tomorrow’s weather (depends on past weather)
• Continue this story (depends on what’s been said)

You need a model that processes inputs one at a time and remembers.

The Vanilla RNN

A Recurrent Neural Network (RNN) processes a sequence step by step:

$$h_t = \tanh(W_h h_{t-1} + W_x x_t + b)$$

• $h_t$ = hidden state at time $t$ (the “memory”)
• $x_t$ = input at time $t$
• $h_{t-1}$ = previous hidden state

At each step:

1. Take current input + previous hidden state
2. Compute new hidden state
3. Optionally output something

Same weights $W_h, W_x$ used at every step — that’s the recurrence.

Visualizing

Unrolled in time:

x_1  x_2  x_3  x_4
 ↓    ↓    ↓    ↓
h_0→h_1→h_2→h_3→h_4
 │    │    │    │
 y_1  y_2  y_3  y_4

The same RNN cell at each step, with memory flowing through $h$.

In PyTorch

import torch.nn as nn

class SimpleRNN(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super().__init__()
        self.rnn = nn.RNN(input_dim, hidden_dim, batch_first=True)
        self.fc = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        # x: (batch, seq_len, input_dim)
        h, _ = self.rnn(x)         # h: (batch, seq_len, hidden_dim)
        return self.fc(h[:, -1])   # use last time step's hidden

The Vanishing Gradient Problem

RNNs were trained with Backpropagation Through Time (BPTT) — backprop through the unrolled sequence.

The problem: gradients multiply through all time steps. For a 50-step sequence:

$$\frac{\partial L}{\partial h_1} = \prod_{t=2}^{50} \frac{\partial h_t}{\partial h_{t-1}}$$

Each factor is < 1 in typical cases → product → 0.

Result: RNNs forget anything more than ~10-20 steps back.

LSTM (1997): The Solution

Sepp Hochreiter & Jürgen Schmidhuber proposed Long Short-Term Memory (LSTM).

Adds a separate cell state $C_t$ that flows through without much modification:

            input gate    forget gate     output gate
              ↓               ↓                ↓
x_t → ────[combine]────→ C_t (cell)─────→ h_t (hidden)
                              ↑
                          C_{t-1}

Three “gates” control what info to:

• Forget (delete from cell)
• Add (write to cell)
• Output (read from cell)

Each gate is a learned sigmoid → values in [0, 1] → multiplicative gating.

LSTM Math

$$\begin{aligned} f_t &= \sigma(W_f \cdot [h_{t-1}, x_t] + b_f) &&\text{forget gate} \\ i_t &= \sigma(W_i \cdot [h_{t-1}, x_t] + b_i) &&\text{input gate} \\ \tilde{C}_t &= \tanh(W_C \cdot [h_{t-1}, x_t] + b_C) &&\text{candidate} \\ C_t &= f_t \odot C_{t-1} + i_t \odot \tilde{C}_t &&\text{cell update} \\ o_t &= \sigma(W_o \cdot [h_{t-1}, x_t] + b_o) &&\text{output gate} \\ h_t &= o_t \odot \tanh(C_t) &&\text{hidden} \end{aligned}$$

Long, but key insight: cell state $C_t$ is updated additively, not multiplicatively → gradient flows.

Gradient can survive 100+ steps. LSTMs could remember much longer.

GRU (2014): Simplification

Gated Recurrent Unit (Cho et al.):

• Combines cell + hidden state into one
• Two gates instead of three
• Faster, similar accuracy

update gate + reset gate → simpler LSTM

Common choice in 2014-2017.

The Golden Age (2014-2017)

LSTMs powered the breakthrough era:

Machine Translation

Sequence-to-sequence (Sutskever 2014):

• Encoder LSTM reads source sentence
• Decoder LSTM generates translation

Google Translate switched to LSTM-based seq2seq in 2016.

Speech Recognition

Baidu DeepSpeech, Google Voice — all LSTM/RNN based.

Image Captioning

CNN reads image → LSTM generates caption.

Text Generation

Early “AI writer” projects — LSTMs trained on books.

The Limits

Despite being a huge improvement over MLPs for sequences, LSTMs had problems:

1. Slow to train

Sequential processing — can’t parallelize across timesteps.

2. Long-range still weak

LSTMs can remember 100+ steps but not 10k. Long documents still hard.

3. Information bottleneck

In seq2seq, all info about the source must squeeze through one vector (the final hidden state).

Long sentences → information loss.

Attention (2015): The Hint of What’s Coming

Bahdanau et al. introduced attention for seq2seq:

• Instead of one final hidden state from encoder
• Decoder can attend to all encoder hidden states
• Different attention weights at different decoding steps

This solved the bottleneck problem. Translation quality jumped.

This was the seed that became the Transformer.

Transformer Kills RNN (2017)

“Attention Is All You Need” (Vaswani et al.) said:

What if we just use attention, no RNN at all?

Result:

• Parallelizable (no sequential dependency)
• Better long-range (attention sees everything)
• Better at every benchmark

By 2019, almost no one was using LSTMs for new research.

What About Today?

RNNs and LSTMs are largely legacy. But:

Still used for:

• Small embedded — RNN cells are tiny
• Streaming inference — naturally process one token at a time
• Tabular time series — sometimes simpler is enough

The renaissance?

Mamba (2024) is a “modernized RNN” — state-space model with linear recurrence. Some papers suggest selective state spaces can match Transformers in some tasks at better efficiency.

So the “RNN is dead” view is partially reversing.

Code: A Tiny LSTM for Text

import torch.nn as nn

class CharLSTM(nn.Module):
    def __init__(self, vocab_size, hidden_dim=128, num_layers=2):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, hidden_dim)
        self.lstm = nn.LSTM(hidden_dim, hidden_dim, num_layers,
                            batch_first=True, dropout=0.3)
        self.fc = nn.Linear(hidden_dim, vocab_size)

    def forward(self, x, hidden=None):
        x = self.embed(x)
        x, hidden = self.lstm(x, hidden)
        return self.fc(x), hidden

# Train on Shakespeare for character-level generation
model = CharLSTM(vocab_size=128, hidden_dim=256, num_layers=2)
# ... training loop ...

Run this for a few hours on Shakespeare → it generates Shakespeare-ish text.

This was how all “text generation” worked in 2015-2017.

A Historical Reflection

The LSTM is one of those algorithms where:

• Math was published 1997
• Adoption took 15 years (waiting for GPU compute)
• Reigned 2014-2017
• Largely displaced by 2019

Total useful life: ~5 dominant years.

But every Transformer paper today references the lineage: MLP → RNN → LSTM → Attention → Transformer → …

The history makes the present make sense.

Next recommended: L3-05 Attention or L3-08 Transformer Architecture.