GAN and VAE: Two Routes to Generative Models

L5-02 covered Diffusion. This piece covers its predecessors — GAN and VAE.

Although Diffusion has won at image generation post-2023, you still need both because:

• GAN remains best in some niches (StyleGAN face details)
• VAE is the foundation of Diffusion (Latent Diffusion = VAE encoder + Diffusion + VAE decoder)

Without understanding these two, you’re just “calling Diffusers.” With them, you understand the entire evolution of generative AI.

One-line definitions

Model	Core idea
VAE (2013)	Compress data into a continuous probabilistic space, then sample to reconstruct
GAN (2014)	Let a generator vs. discriminator play a game — when neither can fool the other, the generator has learned the true distribution
Diffusion (2020)	Start from pure noise and step-by-step denoise into a real image

1. GAN: Adversarial Generation

L5-02 introduced the basic idea. Here we cover the architecture and training details.

Architecture

    Random noise z (Gaussian)
         ↓
    [Generator G] (CNN)
         ↓
     Fake image G(z)
         ↓
                       ←————— Real image x
    [Discriminator D] (CNN)
         ↓
     0 (fake) or 1 (real)

Objective (minimax game)

$$\min_G \max_D \mathbb{E}_{x \sim p_{data}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))]$$

• D wants to maximize: high score for reals, low score for fakes
• G wants to minimize: convince D to give high scores to its fakes

Theoretically the Nash equilibrium has D outputting 0.5 (can’t tell) — at which point G has perfectly captured the real distribution.

Training loop

for batch in real_data:
    # 1. Train discriminator
    real_imgs = batch
    fake_imgs = G(noise)
    d_loss = -log(D(real_imgs)) - log(1 - D(fake_imgs))
    d_loss.backward()
    optimizer_D.step()

    # 2. Train generator
    fake_imgs = G(noise)
    g_loss = -log(D(fake_imgs))  # tries to fool D
    g_loss.backward()
    optimizer_G.step()

GANs are famously hard to train

Historically GAN training is extremely unstable:

Issue	Symptom
Mode collapse	Generator only produces a few samples (no diversity)
Discriminator too strong	D hits 100% accuracy → G has no useful gradient
Training oscillation	Loss bounces around, never converges
Hyperparameter sensitivity	Tiny changes break training

Subsequent work (WGAN, StyleGAN, Progressive GAN) mostly addresses these.

Notable GANs

Model	Breakthrough
DCGAN (2015)	First successful CNN-based GAN, stabilized training
Progressive GAN (2017)	Trains low-res → progressively adds detail
StyleGAN / StyleGAN2 (2018-2019)	SOTA face generation (thispersondoesnotexist)
BigGAN (2018)	Large-scale conditional ImageNet generation
CycleGAN (2017)	Unpaired image translation (horse ↔ zebra)

2. VAE: Probabilistic Compression

Core idea

A neural network learns a “compress + reconstruct” pipeline — but compresses into a probability distribution, not a fixed vector:

Real image x
   ↓
[Encoder] (CNN → MLP)
   ↓
Latent distribution q(z|x) (Gaussian: μ, σ)
   ↓ sample
Latent z
   ↓
[Decoder] (MLP → transposed CNN)
   ↓
Reconstruction x'

Loss = reconstruction + regularization

$$L = \mathbb{E}[\|x - x'\|^2] + D_{KL}(q(z|x) \| p(z))$$

First term: reconstruction should match the input. Second term: latent distribution should match a standard Gaussian (so that “any random vector can generate a plausible image”).

The KL term forces the latent space to be continuous and smooth — which is what distinguishes VAE from a vanilla autoencoder.

Reparameterization trick

Sampling from $q(z|x)$ directly is non-differentiable. Trick:

# Not differentiable:
z = sample(N(mu, sigma))

# Differentiable:
epsilon = sample(N(0, 1))   # doesn't depend on params
z = mu + sigma * epsilon    # this is differentiable!

Move the “randomness” into $\epsilon$ so gradients can flow through $\mu, \sigma$ back to the encoder. This is the key trick that makes VAE trainable.

VAE’s “blurriness” problem

VAE reconstructions are always a bit blurry — a side effect of MSE loss + Gaussian assumption (“average out” different possibilities).

This is why GAN was visually superior to VAE for years.

But VAE’s latent space is more useful: interpolation, semantic editing, etc.

3. GAN vs VAE

Dimension	GAN	VAE
Image quality	Sharp, detailed	Slightly blurry
Training stability	Unstable (mode collapse)	Stable
Controllability	Hard (unstructured latent space)	Strong (continuous, smooth)
Mathematics	Minimax, no clear objective	Variational inference, clean probabilistic framework
Diversity	Easy to lose (mode collapse)	Natural coverage
Generation speed	One forward pass = one image	One forward pass = one image

4. Relationship to Diffusion

Diffusion borrows from both:

• Like VAE: clear probabilistic framework, loss = variational lower bound
• Like GAN: extremely high generation quality
• Better than both: stable training (GAN’s weakness) + sharp images (VAE’s weakness)

Latent Diffusion (the core of Stable Diffusion):

Real → VAE encoder → latent → Diffusion → latent → VAE decoder → Real

The VAE is still inside Diffusion, as the compress/decompress component — not obsolete, just integrated.

5. When you’d still pick GAN / VAE

GANs still win at

• StyleGAN family: high-fidelity faces / single-class generation
• GAN-based super-resolution (ESRGAN)
• CycleGAN-style image translation (still SOTA in some niches)
• Real-time generation (one forward pass vs. Diffusion’s many steps)

VAEs still win at

• Latent space operations: style transfer, attribute editing
• Anomaly detection: high reconstruction error = anomaly
• Data compression: as a learnable compressor
• The latent component of any Diffusion / Flow-based model

A minimal VAE

import torch
import torch.nn as nn

class VAE(nn.Module):
    def __init__(self, input_dim=784, latent_dim=20):
        super().__init__()
        # Encoder
        self.fc1 = nn.Linear(input_dim, 400)
        self.fc_mu = nn.Linear(400, latent_dim)
        self.fc_logvar = nn.Linear(400, latent_dim)
        # Decoder
        self.fc2 = nn.Linear(latent_dim, 400)
        self.fc3 = nn.Linear(400, input_dim)

    def encode(self, x):
        h = torch.relu(self.fc1(x))
        return self.fc_mu(h), self.fc_logvar(h)

    def reparameterize(self, mu, logvar):
        std = torch.exp(0.5 * logvar)
        eps = torch.randn_like(std)
        return mu + eps * std

    def decode(self, z):
        h = torch.relu(self.fc2(z))
        return torch.sigmoid(self.fc3(h))

    def forward(self, x):
        mu, logvar = self.encode(x.view(-1, 784))
        z = self.reparameterize(mu, logvar)
        x_recon = self.decode(z)
        return x_recon, mu, logvar

def vae_loss(x, x_recon, mu, logvar):
    recon = nn.functional.binary_cross_entropy(x_recon, x.view(-1, 784), reduction='sum')
    kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    return recon + kl

50 lines — basic generation on MNIST.

Next: L5-02 Diffusion math or the GAN training visualization.