Generative Adversarial Networks (GAN)

Definition:

GAN Objective

min⁡Gmax⁡D  Ex∼pdata[log⁡D(x)]+Ez∼p(z)[log⁡(1−D(G(z)))]\min_G \max_D \; \mathbb{E}_{\mathbf{x} \sim p_{\text{data}}}[\log D(\mathbf{x})] + \mathbb{E}_{\mathbf{z} \sim p(\mathbf{z})}[\log(1 - D(G(\mathbf{z})))]ThegeneratorThe generatorGmapsnoisetosamples;thediscriminatormaps noise to samples; the discriminatorDclassifiesrealvsfake.Atequilibrium,classifies real vs fake. At equilibrium,D(\mathbf{x}) = 0.5forallfor all\mathbf{x}$.

Definition:

Spectral Normalisation

Constrains the Lipschitz constant of the discriminator by normalising each weight matrix by its spectral norm:

W^=W/σ(W)\hat{\mathbf{W}} = \mathbf{W} / \sigma(\mathbf{W})

from torch.nn.utils import spectral_norm
disc_conv = spectral_norm(nn.Conv2d(64, 128, 3, padding=1))

Example: DCGAN Generator

Implement a DCGAN generator using ConvTranspose2d.

Solution
Implementation
class Generator(nn.Module):
    def __init__(self, z_dim=100, ch=64):
        super().__init__()
        self.net = nn.Sequential(
            nn.ConvTranspose2d(z_dim, ch*4, 4, 1, 0, bias=False),
            nn.BatchNorm2d(ch*4), nn.ReLU(True),
            nn.ConvTranspose2d(ch*4, ch*2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ch*2), nn.ReLU(True),
            nn.ConvTranspose2d(ch*2, ch, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ch), nn.ReLU(True),
            nn.ConvTranspose2d(ch, 1, 4, 2, 1, bias=False),
            nn.Tanh())
    def forward(self, z):
        return self.net(z.view(-1, z.size(1), 1, 1))

GAN Training Dynamics

Watch generator and discriminator losses during training.

Parameters
Variational Autoencoder (VAE)Denoising Diffusion (DDPM)