Information Theory: Entropy and Cross-Entropy

Every classification task uses cross-entropy loss. Why? Because of Shannon’s information theory.

Information = Surprise

Shannon (1948) asked: how do you quantify “information”?

His answer: information equals surprise.

• Telling you “the sun rose this morning” → 0 information (known)
• Telling you “it snowed in Singapore” → high information (surprising)

Mathematically:

$$I(x) = -\log_2 P(x)$$

• $P(x) = 1$ (certain) → $I = 0$
• $P(x) = 0.5$ (coin flip) → $I = 1$ bit
• $P(x) = 0.01$ (rare) → $I ≈ 6.6$ bits

Lower probability event = more information bits.

Entropy: Average Information

Entropy is the expected information:

$$H(P) = -\sum_x P(x) \log_2 P(x)$$

For a fair coin: $H = -0.5 \log_2 0.5 - 0.5 \log_2 0.5 = 1$ bit. For a biased coin (99/1): $H ≈ 0.08$ bit (mostly predictable).

Maximum entropy = uniform distribution (most uncertainty).

Cross-Entropy: Distance Between Distributions

Suppose:

• True distribution: $P$ (label, one-hot like [0, 0, 1, 0])
• Predicted distribution: $Q$ (model output, like [0.1, 0.2, 0.6, 0.1])

Cross-entropy is:

$$H(P, Q) = -\sum_x P(x) \log Q(x)$$

For our example (true class is index 2):

$$H(P, Q) = -(0 \cdot \log 0.1 + 0 \cdot \log 0.2 + 1 \cdot \log 0.6 + 0 \cdot \log 0.1) = -\log 0.6 ≈ 0.51$$

Lower when $Q$ matches $P$. Zero when prediction is perfectly confident on the right answer.

Why It’s the Right Loss

Cross-entropy has properties that make it ideal for classification:

1. Penalizes confident wrong answers

If true class is 2 and model predicts [0.99, 0.001, 0.005, 0.005]: $H = -\log(0.005) ≈ 7.6$ — huge penalty.

If it predicts [0.4, 0.3, 0.2, 0.1] (less confident): $H = -\log(0.2) ≈ 2.3$ — moderate penalty.

Confidence on wrong answers = big penalty.

2. Differentiable

Smooth, so gradient descent works. Compare to “accuracy” which is non-differentiable.

3. Maximum likelihood estimation

Minimizing cross-entropy ≡ maximizing $\log P(\text{data} | \text{model})$ — it’s the MLE objective.

4. KL divergence

Cross-entropy = entropy of P + KL divergence between P and Q:

$$H(P, Q) = H(P) + D_{KL}(P \| Q)$$

Since $H(P)$ is constant (true distribution doesn’t change), minimizing $H(P, Q)$ = minimizing $D_{KL}(P \| Q)$ — making Q close to P.

Binary Cross-Entropy

For binary classification (one output):

$$\text{BCE}(y, \hat{y}) = -y \log \hat{y} - (1-y) \log(1-\hat{y})$$

In PyTorch:

import torch.nn.functional as F

y = torch.tensor([1.0, 0.0, 1.0])  # true labels
y_hat = torch.tensor([0.9, 0.1, 0.4])  # predictions
loss = F.binary_cross_entropy(y_hat, y)

Categorical Cross-Entropy (Multi-Class)

For K classes:

$$\text{CE}(y, \hat{y}) = -\sum_{k=1}^K y_k \log \hat{y}_k$$

In PyTorch (note: takes raw logits, applies softmax internally):

import torch.nn.functional as F

logits = torch.tensor([[2.0, 1.0, 0.1]])  # raw model output
labels = torch.tensor([0])  # class index
loss = F.cross_entropy(logits, labels)

Connection to Information Compression

Shannon’s original framing: cross-entropy = average bits needed to encode P using a code optimized for Q.

If you have model Q that thinks “the” is super common but actual distribution P uses “the” less often — your code is inefficient. The “extra bits” you waste = cross-entropy minus true entropy.

This is why LLMs are measured by perplexity = $2^{H(P, Q)}$ — average “branching factor” the model thinks each token has.

A Concrete Example

Train a sentiment classifier:

# Model output for input "This movie is great"
logits = [3.5, 0.2]  # [positive, negative]
probs = softmax(logits) = [0.96, 0.04]

# True label: positive (class 0)
y = [1, 0]

# Cross-entropy
loss = -(1 * log(0.96) + 0 * log(0.04)) = -log(0.96) ≈ 0.04

Low loss — model is confident in the right answer.

If model had said [0.5, 0.5]: loss = -log(0.5) ≈ 0.69 — much higher.

Backprop pushes the model toward the lower loss.

Practical Tips

1. Don’t combine softmax + cross-entropy yourself

# Bad: numerical instability
probs = softmax(logits)
loss = -y * log(probs)

# Good: PyTorch's cross_entropy handles it
loss = F.cross_entropy(logits, labels)

2. Watch for label smoothing

# Standard: one-hot target [0, 0, 1, 0]
# Smoothed: [0.025, 0.025, 0.925, 0.025] (less confident target)

loss_fn = nn.CrossEntropyLoss(label_smoothing=0.1)

Helps prevent over-confidence (used in image models, modern LLMs).

3. Class imbalance

If 99% of examples are class A:

weights = torch.tensor([0.01, 0.99])  # inverse frequency
loss_fn = nn.CrossEntropyLoss(weight=weights)

Up-weight the rare class so it isn’t ignored.

Next recommended: L1-06 Backpropagation or L1-04 Probability.