Legacy Concept Lab

Loss Landscapes, Sharpness & Flat Minima

Frontier models rely on implicit flat-minima bias (mini-batch SGD, data augmentation, weight decay) for generalization

Concept 4 of 100OptimizationPhase 3

#4SharpnessOptimization

key equation

\min_w \max_{\|\epsilon\|_p \le \rho} L(w + \epsilon)

Phase 3: Optimization & generalizationConcept 4 of 100

Why It Matters for Modern Models

Frontier models rely on implicit flat-minima bias (mini-batch SGD, data augmentation, weight decay) for generalization
Fine-tuning and RLHF pipelines sometimes adopt SAM-like ideas to stabilize training

What is still poorly explained in textbooks and papers:

Most expositions show "sharp vs flat" in 2D, but we lack intuitive stories for high-dimensional anisotropic sharpness
How mode connectivity (many minima connected by low-loss paths) interacts with flatness and weight averaging

If you want intuition first, start with the key equation and the visualization. Come back here for the full walkthrough.

Key Equation

\min_w \max_{\|\epsilon\|_p \le \rho} L(w + \epsilon)

SAM objective:

\min_w \max_{\|\epsilon\|_p \le \rho} L(w + \epsilon)

In practice:

Take a single gradient step to find "worst-case" perturbation: $\epsilon(w) \approx \rho \frac{\nabla L(w)}{\|\nabla L(w)\|_2}$
Update using gradient at the perturbed weights: $\nabla L(w+\epsilon(w))$