Deep Network Training Tricks: BatchNorm / Dropout / Initialization

L3-02 showed how backprop computes gradients. But having gradients isn’t enough to train deep networks —

Before 2012, “training 10+ layers” was a hard research problem.

A series of training tricks finally made deep networks practical. This piece covers the big three.

The Problem: Deep Networks Don’t Train

L3-02 mentioned: chain rule is a product. With sigmoid: Each layer’s local gradient ≤ 0.25. After 10 layers → gradient ≈ 0.0000001. Vanishes.

Classical symptoms:

• First layers don’t learn at all
• Loss flat after 100 epochs
• Tweaking architecture / LR doesn’t help

Solutions in three directions:

1. Better initialization — start gradients in a good range
2. Better activations — ReLU instead of sigmoid
3. Normalization + regularization — BatchNorm, Dropout, etc.

1. Good Initialization

Bad approaches

W = torch.zeros(...)         # all zero → every neuron learns the same thing
W = torch.randn(...) * 1.0   # variance too high → activation explosion

Xavier / Glorot (2010)

Set initial variance so output variance ≈ input variance per layer:

$$W \sim \mathcal{N}\left(0, \frac{2}{n_{\text{in}} + n_{\text{out}}}\right)$$

Good for sigmoid / tanh.

He Initialization (2015)

ReLU kills half of activations → need larger variance:

$$W \sim \mathcal{N}\left(0, \frac{2}{n_{\text{in}}}\right)$$

Default for ReLU networks:

nn.init.kaiming_normal_(layer.weight, mode='fan_in', nonlinearity='relu')

PyTorch defaults to He init for most modules — usually fine.

One-line summary: good initialization keeps gradients alive at the start.

2. ReLU Revolution

Sigmoid’s problem

σ(x) = 1 / (1 + e^(-x))
σ'(x) ≤ 0.25  ← max derivative

10 layers × 0.25 = $10^{-6}$. Vanishing gradient.

ReLU = max(0, x)

ReLU'(x) = 1 (x > 0) or 0 (x ≤ 0)

Positive region gradient is 1 — no decay. Computationally simple. Cheap.

Problem: negative region gradient is 0 → “dying ReLU” (neuron always outputs 0).

Variants

Activation	Notes
Leaky ReLU	Negative side has small slope (0.01x)
GELU	Smooth ReLU, default for Transformers
Swish / SiLU	$x \cdot \sigma(x)$, GLU family
Mish	Smoother variant

Practical: CNN uses ReLU / Leaky ReLU; Transformer uses GELU.

3. Batch Normalization (2015)

Intuition

Each layer’s input distribution keeps shifting during training (because earlier layer parameters change). This is called internal covariate shift.

BatchNorm forces each layer’s activations to a stable distribution.

Formula

For a batch $\{x_1, ..., x_m\}$:

1. Mean: $\mu = \frac{1}{m}\sum x_i$
2. Variance: $\sigma^2 = \frac{1}{m}\sum (x_i - \mu)^2$
3. Normalize: $\hat{x}_i = \frac{x_i - \mu}{\sqrt{\sigma^2 + \epsilon}}$
4. Scale + shift: $y_i = \gamma \hat{x}_i + \beta$ ($\gamma, \beta$ learnable)

The learnable scale/shift is crucial — lets the model undo normalization if needed.

PyTorch

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(100, 256),
    nn.BatchNorm1d(256),    # ← add this
    nn.ReLU(),
    nn.Linear(256, 10),
)

Train vs Inference

• Train: use current batch’s stats $\mu, \sigma^2$
• Inference: use running average from training

model.train()   # uses batch stats
model.eval()    # uses running average

Forgetting to switch to eval is a top bug.

Variants

Method	When to use
BatchNorm	CNN / images
LayerNorm	Transformer / NLP (variable-length sequences)
GroupNorm	Small batch (< 16)
InstanceNorm	Style transfer

Transformers use LayerNorm because batch stats are unstable when sequence lengths vary.

4. Dropout (2012)

Intuition

During training, randomly drop some neurons (set to 0). Forces the network to not depend on any one neuron — learns redundant representations.

Equivalent to:

• Ensemble averaging many slightly-different networks
• Noise injection → robust features

Math

# Training
mask = (random > 0.5)         # 50% dropout
output = input * mask / 0.5    # rescale to compensate

Divide by keep probability to preserve expected value (inverted dropout).

PyTorch

nn.Sequential(
    nn.Linear(100, 256),
    nn.ReLU(),
    nn.Dropout(0.5),       # 50% dropout rate
    nn.Linear(256, 10),
)

Turn Off at Inference

model.eval() automatically disables dropout (no drop + no rescale).

Dropout Rates

Network	Recommended
Fully-connected MLP	0.3-0.5
CNN fully-connected layers	0.5
CNN conv layers	0.1-0.2 or none
Transformer	0.1
Very large model + abundant data	0 (not needed)

Modern LLMs typically don’t use dropout — data scale removes overfitting concern.

5. Gradient Clipping

Brute force against exploding gradients:

torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

If total gradient norm > 1.0 → scale all gradients down proportionally.

Almost always used in RNN / LSTM training. Common in Transformer training too.

6. Residual Connections (2015, ResNet)

Not a single trick, but fundamentally solves vanishing gradient.

Standard layer: $y = F(x)$ Residual layer: $y = F(x) + x$

In backward:

$$\frac{\partial L}{\partial x} = \frac{\partial L}{\partial y} \cdot \left(\frac{\partial F}{\partial x} + 1\right)$$

That +1 — gradient always has a “highway” back, never vanishes.

Allows training 100, 200, even 1000 layers. Every modern architecture (CNN, Transformer) uses residual connections.

A Modern CNN Block

What a standard ResNet block looks like:

class ResBlock(nn.Module):
    def __init__(self, channels):
        super().__init__()
        self.conv1 = nn.Conv2d(channels, channels, 3, padding=1)
        self.bn1 = nn.BatchNorm2d(channels)
        self.conv2 = nn.Conv2d(channels, channels, 3, padding=1)
        self.bn2 = nn.BatchNorm2d(channels)

    def forward(self, x):
        residual = x                      # ← residual path
        out = self.bn1(self.conv1(x))
        out = torch.relu(out)
        out = self.bn2(self.conv2(out))
        out = out + residual              # ← combine
        return torch.relu(out)

Contains: He init (default), ReLU, BatchNorm, residual. Standard recipe for modern deep learning.

Training Checklist

When a new network doesn’t train:

1. ✅ ReLU / GELU for hidden layers (not sigmoid)
2. ✅ He / Xavier init (PyTorch defaults usually fine)
3. ✅ BatchNorm / LayerNorm added
4. ✅ Residual connections for deep nets
5. ✅ Gradient clipping for RNN / large models
6. ✅ Dropout 0.3-0.5 in FC layers
7. ✅ Try learning rate 1e-3 first, scale up/down
8. ✅ Loss doesn’t decrease → check data / labels
9. ✅ Train loss low + val high → more regularization / augmentation

Architectural Choice Matters

Modern alternatives to “BN + ReLU”:

Combo	Used in
BatchNorm + ReLU	Classic CNN (ResNet, VGG)
LayerNorm + GELU	Transformer
RMSNorm + SiLU	LLaMA, modern LLM

Each generation refines what works best for the specific architecture.

Next recommended: L3-04 RNN / LSTM or L3-07 Attention Variants.