L3 Chapter 2 🐣 🕒 9 min

Backpropagation Deep Dive: Chain Rule in Action

Backprop is what makes neural networks actually learn. Unpacking the math, intuition, and implementation.

Alai

9/2/2026

L3-01 showed how to stack MLP layers. This piece: how stacked layers actually “learn”.

Without backprop, neural networks are just expensive random functions.

This piece is the deep version of L1-06.

One-Sentence Definition

Backprop = use the chain rule to propagate the loss’s gradient back through every layer’s parameters.

Each layer gets its own “you should go in this direction, this much” signal — and gradient descent walks them in that direction.

Math Refresher: Chain Rule

L1-06 covered this. Quick recap:

\frac{d}{dx} f(g(x)) = f'(g(x)) \cdot g'(x)

Derivative of a composition = derivative of outer × derivative of inner.

Generalize to many variables, many layers → backpropagation.

A Minimal Neural Network

1 input, 1 hidden layer (2 neurons), 1 output:

x ──W1──> h ──W2──> ŷ ──Loss──> L
                              ↑
                              y

Parameters: $W_1, W_2$ (ignore biases for simplicity) Activation: $h = \sigma(W_1 x)$ , sigmoid Prediction: $\hat{y} = W_2 h$ Loss: $L = \frac{1}{2}(\hat{y} - y)^2$

Question: what’s $\frac{\partial L}{\partial W_1}$ ?

Computing It Layer by Layer

Apply chain rule, outer to inner:

\frac{\partial L}{\partial W_1} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial h} \cdot \frac{\partial h}{\partial W_1}

Each piece is a local derivative:

$\frac{\partial L}{\partial \hat{y}} = \hat{y} - y$ (loss → prediction)
$\frac{\partial \hat{y}}{\partial h} = W_2$ (linear layer)
$\frac{\partial h}{\partial W_1} = \sigma'(W_1 x) \cdot x$ (activated weight)

Put together:

\frac{\partial L}{\partial W_1} = (\hat{y} - y) \cdot W_2 \cdot \sigma'(W_1 x) \cdot x

Simple, every piece is local microdifferentiation.

”Backward” Direction

Forward: $x \to h \to \hat{y} \to L$ Backward: $L \to \hat{y} \to h \to x$ (and the gradients along the way)

Forward:  x → h → ŷ → L
Backward: ∂L/∂x ← ∂L/∂h ← ∂L/∂ŷ ← 1

Gradients flow backwards through the network — that’s the “back” in backprop.

Why Not Just Use Numerical Differentiation

Finite differences:

\frac{\partial L}{\partial W_1} \approx \frac{L(W_1 + \epsilon) - L(W_1)}{\epsilon}

For 100M parameters? 100M forward passes per gradient.

Backprop: one forward + one backward, get all 100M gradients.

Numerical: O(N) forward passes per gradient. Backprop: O(1) forward passes per gradient.

This is what makes deep learning feasible.

Computational Graph

Modern frameworks (PyTorch, TF) represent the network as a directed acyclic graph:

       ┌──────┐
       │  x   │
       └──┬───┘
          │
       ┌──▼───┐    ┌────┐
       │ mul  │◄───│ W1 │
       └──┬───┘    └────┘
          │
       ┌──▼───┐
       │sigma │
       └──┬───┘
          │
       ┌──▼───┐    ┌────┐
       │ mul  │◄───│ W2 │
       └──┬───┘    └────┘
          │
       ┌──▼───┐
       │ ŷ   │
       └──┬───┘
          │
       ┌──▼───┐    ┌────┐
       │ loss │◄───│ y  │
       └──────┘    └────┘

Each node knows:

Forward rule
Backward rule (given gradient on output, compute gradients on inputs + parameters)

Backward pass = traverse the graph in reverse.

In PyTorch

PyTorch’s autograd builds + traverses the graph automatically:

import torch

x = torch.tensor(2.0)
y = torch.tensor(1.0)

W1 = torch.tensor(0.5, requires_grad=True)
W2 = torch.tensor(0.8, requires_grad=True)

# Forward
h = torch.sigmoid(W1 * x)
y_pred = W2 * h
loss = 0.5 * (y_pred - y) ** 2

print(f"loss = {loss.item():.4f}")

# Backward — one line
loss.backward()

print(f"dL/dW1 = {W1.grad:.4f}")
print(f"dL/dW2 = {W2.grad:.4f}")

loss.backward() does the entire chain-rule internally.

Common Pitfalls

1. Gradients Accumulate

loss.backward()
# W1.grad is 0.3

loss.backward()
# W1.grad is 0.6 — accumulated!

Training loops MUST:

optimizer.zero_grad()   # clear previous gradients
loss.backward()         # new gradients
optimizer.step()        # apply

Forgetting zero_grad is a top new-developer bug.

2. Vanishing / Exploding Gradients

Chain rule = product. Through many layers:

All factors < 1 → product → 0 (vanishing)
All factors > 1 → product → ∞ (exploding)

Classical symptoms:

Sigmoid + deep net → vanishing gradient → can’t train deep
RNN with long sequence → both happen

Solutions (covered L3-03):

ReLU activations
BatchNorm / LayerNorm
Residual connections (ResNet)
Gradient clipping

3. Non-Differentiable Ops

argmax, if x > 0, sampling → no gradient.

Workarounds:

Gumbel-Softmax: continuous relaxation of argmax
Straight-Through Estimator: forward discrete, backward identity-like
Reparameterization (VAE-style): move randomness outside

Historical Note

1960s: similar ideas in cybernetics, control theory
1986: Rumelhart, Hinton, Williams formalize for neural nets
2012: AlexNet uses GPUs + backprop on ImageNet → deep learning explodes
Today: autograd is “free” for every PyTorch user

Backprop is one of the most important algorithms in AI history. Without it, “training neural networks” wouldn’t be a thing.

Implementation Detail: Reverse-Mode Autodiff

PyTorch implements “reverse-mode automatic differentiation”:

# Forward pass: build the graph (record operations)
y = f(x)  # ← every op records itself in the graph

# Backward pass:
# 1. Start from output, gradient = 1
# 2. Walk graph backward
# 3. For each node, use its "backward" rule to compute input gradients
# 4. Accumulate parameter gradients along the way

Reverse-mode is efficient when there are many inputs (parameters) and few outputs (loss) — which is exactly the ML case (millions of params, scalar loss).

The opposite (forward-mode) is better when you have few params and many outputs — used in physics, control.

A Worked Example

Compute gradient by hand:

$f(x) = (x^2 + 3x + 1)^2 \text{ at } x = 2$

Inner: $g(x) = x^2 + 3x + 1 = 4 + 6 + 1 = 11$ Outer: $f = g^2 = 121$

Derivative: $\frac{df}{dx} = 2g \cdot \frac{dg}{dx} = 2 \cdot 11 \cdot (2x + 3) = 22 \cdot 7 = 154$

Verify with PyTorch:

x = torch.tensor(2.0, requires_grad=True)
inner = x**2 + 3*x + 1
f = inner**2
f.backward()
print(x.grad)  # 154

This is the actual math behind every training step.

When Backprop Doesn’t Work

There are cases where backprop is theoretically not enough:

Reinforcement Learning: reward is non-differentiable through environment
Discrete decisions: argmax can’t backprop through
Black-box optimization: function isn’t expressible as a graph

These need different methods (REINFORCE, evolution strategies, etc.).

What About Forward-Mode?

For “small input → big output” (e.g., Jacobian-vector products in physics simulations):

Forward-mode: compute as you go forward, more memory-efficient for narrow nets
Reverse-mode (our backprop): better for ML’s “many parameters” setup

Modern PyTorch supports both via torch.func.jvp and torch.func.vjp.

💡 The take-away

Backprop is chain rule + computational graph + gradients flow backwards.

You don’t have to derive backward formulas by hand (autograd does it) — but understanding what’s happening is essential:

Each layer gets a signal: “go in this direction”
The signal’s quality (not vanishing, not exploding, not too noisy) determines if training works
This is why all the L3-03 tricks matter (BatchNorm, residual, init)

Master backprop → everything else makes sense.

Next recommended: L3-03 Training Tricks or L3-04 RNN / LSTM.