Probability Distributions: Gaussian / Bernoulli / Poisson / Exponential / Categorical

L1-04 covered the basics of probability. This piece covers what probability “looks like” — the 5 distributions you’ll meet most often in ML.

Why distributions matter

Every ML task implicitly assumes data comes from some distribution:

• Regression → Gaussian
• Binary classification → Bernoulli
• Multi-class → Categorical (a.k.a. multinomial)
• Counts → Poisson
• Waiting times → Exponential

If you don’t understand distributions, you’re just “calling library functions.” Once you do, you start to understand what the model is actually doing.

1. Bernoulli

The simplest one — a single trial with two outcomes (success/failure, positive/negative, click/no-click):

$$P(X=1) = p, \quad P(X=0) = 1-p$$

One parameter, $p \in [0, 1]$.

In ML:

• Binary labels: $y \in \{0, 1\}$
• Logistic regression output = the Bernoulli parameter $p$
• The cross-entropy loss is the negative log-likelihood of this distribution

import numpy as np
samples = np.random.binomial(n=1, p=0.7, size=1000)
print(samples.mean())  # ~0.7

2. Categorical / Multinomial

The multi-class generalization of Bernoulli — K possible outcomes per trial:

$$P(X=k) = p_k, \quad \sum_k p_k = 1$$

In ML:

• Multi-class labels (softmax outputs are Categorical parameters)
• A language model predicting the next token = one Categorical sample
• Multi-class cross-entropy loss is the negative log-likelihood of this distribution

# Given probs [0.1, 0.7, 0.2], sample from 3 classes
samples = np.random.choice(3, size=1000, p=[0.1, 0.7, 0.2])

3. Gaussian / Normal

The most important distribution in ML. The bell curve:

$$p(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)$$

Two parameters: mean $\mu$, variance $\sigma^2$.

Why so common:

• Central Limit Theorem: a sum of many independent random variables converges to a Gaussian
• Mathematically convenient (analytic derivatives, conjugate priors, etc.)
• Many real-world noise sources are approximately Gaussian (measurement error, human heights, etc.)

In ML:

• Linear regression assumes Gaussian residuals
• Weight initialization (Xavier, He) is Gaussian
• VAE / Diffusion latents are Gaussian
• MSE loss is equivalent to “assume noise is Gaussian and use the negative log-likelihood”

samples = np.random.normal(loc=0, scale=1, size=1000)
print(samples.mean(), samples.std())  # ~0, ~1

Multivariate Gaussian

The 2D-and-up version:

$$p(\mathbf{x}) = \frac{1}{\sqrt{(2\pi)^d |\Sigma|}} \exp\left(-\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu})^T \Sigma^{-1}(\mathbf{x}-\boldsymbol{\mu})\right)$$

The covariance matrix $\Sigma$ encodes correlations between dimensions. This is the foundation for Gaussian Mixture Models, Kalman filters, and more.

4. Poisson

Count of events in a fixed interval:

$$P(X=k) = \frac{\lambda^k e^{-\lambda}}{k!}$$

Parameter $\lambda$ is the mean event rate.

Typical use cases:

• Number of daily website visitors
• Phone calls arriving at a call center per minute
• “Rare event counts” in scientific experiments

In ML:

• Implicit-feedback modeling in recommender systems
• Count prediction (clicks, conversions)
• A/B test hypothesis testing for low-frequency events

samples = np.random.poisson(lam=3, size=1000)
print(samples.mean())  # ~3

5. Exponential

Waiting time — when does the next event occur:

$$p(x) = \lambda e^{-\lambda x}, \quad x \geq 0$$

Typical use cases:

• Time between customer arrivals
• Time between equipment failures
• “Uptime without a bug” after a deploy

Relationship to Poisson: if event counts are Poisson with rate $\lambda$, then the inter-event times are Exponential with the same $\lambda$.

samples = np.random.exponential(scale=2, size=1000)
print(samples.mean())  # ~2 (scale = 1/lambda)

Quick reference

Distribution	Type	Params	One-line use
Bernoulli	Discrete	$p$	A single binary outcome
Categorical	Discrete	$p_1, ..., p_K$	A single multi-class outcome
Gaussian	Continuous	$\mu, \sigma^2$	Continuous values + default noise assumption
Poisson	Discrete	$\lambda$	Event counts in a fixed interval
Exponential	Continuous	$\lambda$	Waiting time until next event

Sampling in PyTorch

import torch

# Gaussian
torch.distributions.Normal(loc=0, scale=1).sample((1000,))

# Bernoulli
torch.distributions.Bernoulli(probs=0.7).sample((1000,))

# Categorical
torch.distributions.Categorical(probs=torch.tensor([0.1, 0.7, 0.2])).sample((1000,))

# Poisson
torch.distributions.Poisson(rate=3.0).sample((1000,))

3 common pitfalls

1. Treating Gaussian as universal — heavy-tailed data (incomes, network latencies) is severely under-estimated for extreme values by a Gaussian. Consider log-normal, Student’s t, or Pareto.
2. Too few samples to see the shape — you need at least 30+ samples and a QQ-plot to make any judgment about “is this Gaussian?”
3. Singular covariance matrix in multivariate Gaussian — common when dimensions > samples. Regularize or reduce dimensions via PCA.

Picking a distribution for your modeling assumption

My data is	Use
Binary (0/1)	Bernoulli
Multi-class (1…K)	Categorical
Continuous, single peak, symmetric	Gaussian
Continuous, heavy-tailed	Student’s t / Pareto
Counts (0, 1, 2, …)	Poisson
Waiting times	Exponential / Gamma
Proportions (0-1)	Beta
Strictly positive	Log-normal / Gamma

Next: L1-12 Hypothesis testing + A/B testing or L1-05 Information theory.