Continuous Function

Legacy Concept Lab

Adam & Adaptive Gradient Methods

Large foundation models almost universally use Adam or AdamW for pretraining and fine-tuning

Concept 3 of 100OptimizationPhase 3
Open New NotebookAll Foundations
#3AdamOptimization
key equation
θt+1=θt−αm^tv^t+ε\theta_{t+1} = \theta_t - \alpha \frac{\hat m_t}{\sqrt{\hat v_t} + \varepsilon}
Phase 3: Optimization & generalizationConcept 3 of 100
Migrated:view the updated version in /domains →This /foundations page is legacy during migration.

Why It Matters for Modern Models

  • Large foundation models almost universally use Adam or AdamW for pretraining and fine-tuning
  • RLHF and diffusion training use Adam-style optimizers to handle noisy gradients and widely varying scales

What Tutorials Skip

What is still poorly explained in textbooks and papers:

  • Geometric explanation of how per-coordinate scaling with 1/√vₜ interacts with overparameterized nets—why it sometimes hurts generalization vs SGD
  • How Adam bias-correction and exponential averaging interact with curriculum and non-stationary objectives (e.g. RLHF)

Interactive Visualization

Core Math (Optional Deep Dive)

If you want intuition first, start with the key equation and the visualization. Come back here for the full walkthrough.

Key Equation
θt+1=θt−αm^tv^t+ε\theta_{t+1} = \theta_t - \alpha \frac{\hat m_t}{\sqrt{\hat v_t} + \varepsilon}

For gradient gt=∇θLt(θt)g_t = \nabla_\theta L_t(\theta_t):

mt=β1mt−1+(1−β1)gtvt=β2vt−1+(1−β2)gt2m^t=mt/(1−β1t),v^t=vt/(1−β2t)θt+1=θt−αm^tv^t+ε\begin{aligned} m_t &= \beta_1 m_{t-1} + (1-\beta_1) g_t \\ v_t &= \beta_2 v_{t-1} + (1-\beta_2) g_t^2 \\ \hat m_t &= m_t / (1-\beta_1^t),\quad \hat v_t = v_t / (1-\beta_2^t) \\ \theta_{t+1} &= \theta_t - \alpha \frac{\hat m_t}{\sqrt{\hat v_t} + \varepsilon} \end{aligned}

Convergence analyses show that naïve Adam can diverge on simple convex problems and motivate variants like AMSGrad.

Canonical Papers

Adam: A Method for Stochastic Optimization

Kingma & Ba2014ICLR
Read paper →

On the Convergence of Adam and Beyond

Reddi et al.2018ICLR
Read paper →

Connections

Enables

⌇Sharpness📉LR Schedules🧭Natural Grad⚖️AdamW

Next Moves

Explore this concept from different angles — like a mathematician would.

Semantic Connections

⚠️ Breaks When
⌇Edge of Stability→Sharpness⌇Edge of stability→Sharpness
🔄 Same Technique
⚡Adaptive step sizing→Efficiency
🔧 Invented to Fix
ℒEfficient MLE→ML/CE/KL⚔TTUR training→GANs