Legacy Concept Lab

GANs & Adversarial Divergence Minimization

Adversarial min-max ideas appear in adversarial training and some alignment techniques

Concept 8 of 100Generative ModelsPhase 4

Open Demo All Foundations

#8GANsGenerative Models

key equation

\min_G \max_D \; \mathbb E_{p_{\text{data}}}[\log D(x)] + \mathbb E_{p(z)}[\log(1 - D(G(z)))]

Phase 4: Generative modeling familiesConcept 8 of 100

Why It Matters for Modern Models

Adversarial min-max ideas appear in adversarial training and some alignment techniques
GAN-like training still influential in high-fidelity image/video generation

What Tutorials Skip

What is still poorly explained in textbooks and papers:

Why JS divergence leads to vanishing gradients when supports don't overlap, and how Wasserstein distances fix this
Geometric visualizations of discriminator decision surfaces over latent manifolds

Interactive Visualization

Core Math (Optional Deep Dive)

If you want intuition first, start with the key equation and the visualization. Come back here for the full walkthrough.

Key Equation

\min_G \max_D \; \mathbb E_{p_{\text{data}}}[\log D(x)] + \mathbb E_{p(z)}[\log(1 - D(G(z)))]

Original GAN objective:

\min_G \max_D \; \mathbb E_{x\sim p_{\text{data}}}[\log D(x)] + \mathbb E_{z\sim p(z)}[\log(1 - D(G(z)))]

At optimum, with optimal discriminator $D^*$ , this minimizes the Jensen–Shannon divergence between model and data.

WGAN replaces JS with Earth-Mover (Wasserstein-1) distance, with Lipschitz constraints on $D$ .

Canonical Papers

Generative Adversarial Nets

Goodfellow et al.2014NeurIPS

Wasserstein GAN

Arjovsky et al.2017ICML

Connections

Prerequisites

Enables

⚖️OT/Wasserstein

Next Moves

Explore this concept from different angles — like a mathematician would.

Semantic Connections

↔️ Mathematical Dual

∂Adversarial ↔ Denoising→Diffusion ℤImplicit vs explicit density→VAEs

≈ Analogy

⚖Learned critic→RLHF ⌇Saddle point game→Sharpness

🔄 Same Technique

⚠️Learned loss gaming→Reward Hacking ℒMLE-trained classifier→ML/CE/KL

⚠️ Breaks When

ℒNon-overlapping supports→ML/CE/KL

🔧 Invented to Fix

∇TTUR training→Adam ∂Fix mode collapse→Diffusion