Backpropagation: The Chain Rule, in Action

If you only learn one thing about how neural networks train, learn backpropagation.

Backprop turns “I have a giant function with millions of parameters” into “I can compute every parameter’s gradient in one efficient pass”.

The Setup

You have a function $f(x; W_1, W_2, ..., W_L)$ — a deep neural network.

You compute a loss $L$.

You want to find: for each parameter $W_i$, how should I change it to decrease $L$?

You need $\frac{\partial L}{\partial W_i}$ for every $i$.

Naive Approach (Doesn’t Scale)

Numerical differentiation:

$$\frac{\partial L}{\partial W_i} \approx \frac{L(W_i + \epsilon) - L(W_i)}{\epsilon}$$

For each parameter, do one extra forward pass.

100M parameters = 100M forward passes per training step. Impractical.

Backprop: The Idea

Compute all gradients in two passes (one forward, one backward), not 100M passes.

The trick: chain rule + dynamic programming.

Chain Rule Refresher

If $y = f(g(x))$:

$$\frac{dy}{dx} = \frac{dy}{dg} \cdot \frac{dg}{dx}$$

The derivative of a composite function = derivative of outer × derivative of inner.

Backprop in a Tiny Network

Consider: $x \to h = \sigma(W_1 x) \to \hat{y} = W_2 h \to L = \frac{1}{2}(\hat{y} - y)^2$

Question: what’s $\frac{\partial L}{\partial W_1}$?

By chain rule:

$$\frac{\partial L}{\partial W_1} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial h} \cdot \frac{\partial h}{\partial W_1}$$

Each piece is simple:

• $\frac{\partial L}{\partial \hat{y}} = \hat{y} - y$ (loss derivative)
• $\frac{\partial \hat{y}}{\partial h} = W_2$ (linear layer)
• $\frac{\partial h}{\partial W_1} = \sigma'(W_1 x) \cdot x$ (sigmoid + multiplication)

So:

$$\frac{\partial L}{\partial W_1} = (\hat{y} - y) \cdot W_2 \cdot \sigma'(W_1 x) \cdot x$$

That’s a complete, computable expression.

The “Backward” in Backprop

Visualize:

Forward:  x → h → ŷ → L
Backward: ∂L/∂x ← ∂L/∂h ← ∂L/∂ŷ ← 1

Gradients flow backwards through the network — propagating from the loss to each parameter.

This is why it’s called backpropagation.

Computational Graph

Modern frameworks (PyTorch, TensorFlow) represent the network as a graph:

       ┌──────┐
       │  x   │
       └──┬───┘
          │
       ┌──▼───┐    ┌────┐
       │ mul  │◄───│ W1 │
       └──┬───┘    └────┘
          │
       ┌──▼───┐
       │sigma │
       └──┬───┘
          │
       ┌──▼───┐    ┌────┐
       │ mul  │◄───│ W2 │
       └──┬───┘    └────┘
          │
       ┌──▼───┐
       │  ŷ   │
       └──┬───┘
          │
       ┌──▼───┐    ┌────┐
       │ loss │◄───│ y  │
       └──────┘    └────┘

Each node knows:

1. Forward: compute output from inputs
2. Backward: given $\partial L / \partial \text{output}$, compute $\partial L / \partial \text{input}$ and $\partial L / \partial \text{parameter}$

Full backward = walking the graph in reverse.

Why It’s Fast

For an L-layer network with N parameters:

• Naive: O(N × L) operations
• Backprop: O(L) operations

Backprop is N times faster than naive.

Plus, computations are shared — intermediate results from forward pass are reused.

PyTorch Autograd

PyTorch builds the graph automatically:

import torch

# Input and target
x = torch.tensor(2.0)
y = torch.tensor(1.0)

# Parameters — requires_grad=True lets autograd track
W1 = torch.tensor(0.5, requires_grad=True)
W2 = torch.tensor(0.8, requires_grad=True)

# Forward
h = torch.sigmoid(W1 * x)
y_pred = W2 * h
loss = 0.5 * (y_pred - y) ** 2

print(f"loss = {loss.item():.4f}")

# Backward — one line!
loss.backward()

# Gradients live on parameters
print(f"dL/dW1 = {W1.grad:.4f}")
print(f"dL/dW2 = {W2.grad:.4f}")

loss.backward() does the entire chain rule internally.

Common Pitfalls

1. Gradient accumulation

loss.backward()
# W1.grad is 0.3

loss.backward()
# W1.grad is 0.6 — accumulated!

So training loops must:

optimizer.zero_grad()   # Clear previous gradients
loss.backward()         # Compute new gradients
optimizer.step()        # Update along gradient

Forgetting zero_grad() is a common new-developer bug.

2. Vanishing / exploding gradients

Chain rule is a product. If every layer’s local gradient is < 1, product becomes ~0 after many layers (vanishing). If > 1, explodes.

Solutions (covered in L3-03):

• ReLU instead of sigmoid
• BatchNorm
• Residual connections
• Gradient clipping

3. Non-differentiable operations

argmax, if, sampling → no derivative.

Workarounds:

• Straight-Through Estimator: forward uses discrete, backward pretends continuous
• Reparameterization: move randomness outside
• Gumbel-Softmax: differentiable argmax approximation

Backprop History

• 1960s: similar ideas in cybernetics
• 1986: Rumelhart, Hinton, Williams formalize for neural nets
• 2012: AlexNet + GPUs + backprop = deep learning explodes
• Today: autograd makes it “free” for every developer

Backprop is arguably the most important algorithm in AI history — without it, training neural networks isn’t feasible.

Try It Yourself

Compute gradients by hand for a tiny network. The math will click.

# Compute gradient of: f(x) = (x^2 + 3x + 1)^2  at x = 2

x = 2.0
inner = x**2 + 3*x + 1   # = 4 + 6 + 1 = 11
f = inner**2              # = 121

# By chain rule:
# df/dx = 2*inner * d(inner)/dx = 2*11 * (2x + 3) = 22 * 7 = 154

# Verify with autograd
import torch
x_t = torch.tensor(2.0, requires_grad=True)
inner = x_t**2 + 3*x_t + 1
f = inner**2
f.backward()
print(x_t.grad)  # Should be 154

Next recommended: L3-01 Perceptron to MLP or L1-05 Information Theory.