Legacy Atlas

Mathematical Foundations

The foundations library is the broad map: 100 interconnected concepts, legacy demos, canonical papers, and migration links into the newer domain notebooks.

100 concepts37 demos336 connections13 learning phases

Open Concept Map Browse Concepts

1Core

2Optim

3Gen

4Rep

5Scale

6Systems

lab surfacepredict, drag, test

How To Use This Layer

Foundations is the wide map; domains are the refined notebooks.

Keep the old map useful while migration continues: browse broadly here, then step into domain notebooks when a concept has a newer treatment.

SurveyUse the graph when you need the map.

The legacy foundations graph is still the fastest way to see prerequisites, hubs, and semantic edges across the full 100-concept atlas.

Open concept map

SequenceUse the study path when order matters.

The phases turn a broad topic list into a teachable route from core training ideas to systems, alignment, and frontier methods.

LabUse demo filters when you want to learn by doing.

The old dark demo surfaces stay intact here while the surrounding experience points learners toward notebooks and domain pages.

Recommended Study Order

Build understanding from fundamentals to frontier techniques. Each phase builds on the previous one.

1

Core probabilistic training + transformers

ℒML/CE/KL ⊗Attention

2

Architecture fundamentals

🔤Tokens ↻RoPE ⚙Efficient Attention 📏LayerNorm ⊕Residuals 🚪SwiGLU 👥GQA

3

Optimization & generalization

⟲Backprop 🏃SGD+Momentum ∇Adam ⚖️AdamW ✂️Grad Clip ⌇Sharpness ∪Double Descent ΘNTK 📉LR Schedules 🎲Init 🎯Label Smooth 📊BatchNorm

4

Generative modeling families

ℤVAEs ⚔GANs ∂Diffusion 🌊Flows ∇pScore Matching 🎯Consistency 🌊Flow Match

5

Representation & interpretability

◎Embeddings ⊕Superposition ⚲Probing ⊛Circuits 🔍Sparse Autoencoders 🔬Circuit Discovery 🎚️Activation Steering 📝ICL 🔬Logit Lens 🔗Contrastive 🔄SSL

6

Modern efficiency & inference

⏩Speculative Decoding ⚡LLM Serving 🔀MoE ⚡MoE Serving 📏Long Context 🔀SSMs & Hybrids 🎲Decoding 🧪Distillation 🔢Quantization ✂️Pruning ⚡FlashAttn 💾Checkpointing

7

Alignment & RLHF

⚖RLHF 🎯DPO 👍KTO ⚠️Reward Hacking 🎯PPO 📜Constitutional 🤖RLAIF 📝Instruct

8

Scaling, theory & multimodal

↗Scaling ⚡Efficiency ∀Theory 🖼️Multimodal VLP ⚖️OT/Wasserstein ⚔️Adversarial 💡Grokking 🌡️Calibration

9

Advanced architectures & generation

🎚️CFG 🔍RAG 🌐Distributed 🌳Beam Search

10

Mathematical foundations & information geometry

ℐFisher Info 🧭Natural Grad ⚡EBMs 💧Dropout 🪞Bregman 🎭RKHS 🕳️TDA 🔀Lie Groups 🌍World Models

11

Frontier research & scaling

🧠Test-Time 💭CoT 🏭Synth Data 📋PRMs

12

Advanced alignment & safety research

🎯Deliberative ⚔️Debate 🔄IDA 💪Weak→Strong 🔴Auto RedTeam 🔮Mesa-Opt 😴Sleepers ⚖️LLM-as-Judge 🔍Elicitation 🥪Sandwich

13

Cutting-edge 2024-2025 research

📊MoD 🌲MCTS-LLM 🎬VideoWM 🔄Self-Improve 📉Collapse ♾️InfCtx

All 100 Concepts

Search and filter to find what you're looking for.

ML/CE/KL

Maximum Likelihood, Cross-Entropy & KL Divergence

Core Training⭐ 261 paper▶ Demo

Attention

Scaled Dot-Product Attention & Transformer Layers

Core Training⭐ 231 paper▶ Demo

Adam

Adam & Adaptive Gradient Methods

Optimization🔗 42 papers▶ Demo

Sharpness

Loss Landscapes, Sharpness & Flat Minima

Optimization⭐ 112 papers▶ Demo

Double Descent

Overparameterization & Generalization, Double Descent

Optimization🔗 42 papers▶ Demo

NTK

Neural Tangent Kernel & Infinite-Width Limits

Theory🔗 41 paper▶ Demo

VAEs

Variational Autoencoders & Variational Inference

Generative Models🔗 51 paper▶ Demo

GANs

GANs & Adversarial Divergence Minimization

Generative Models🔗 22 papers▶ Demo

Diffusion

Diffusion, Score-Based Models & Flow Matching

Generative Models⭐ 114 papers▶ Demo

Embeddings

Representation Learning & Embedding Geometry

Representations⭐ 141 paper▶ Demo

Superposition

Superposition, Sparse Features & Monosemanticity

Representations🔗 32 papers▶ Demo

Probing

Probing, Linear Classifier Probes & Activation Analysis

Representations🔗 52 papers▶ Demo

Circuits

Transformer Circuits, Induction Heads & Mechanistic Interpretability

Representations🔗 72 papers▶ Demo

Scaling

Scaling Laws & Emergent Abilities

Scaling & Alignment🔗 73 papers▶ Demo

RLHF

Preference-Based Alignment: RLHF, Reward Modeling, Constitutional AI

Scaling & Alignment⭐ 173 papers▶ Demo

Efficiency

Efficiency: Quantization, Distillation, LoRA & Sparse MoE

Efficiency🔗 93 papers▶ Demo

Theory

Theoretical Foundations: PAC Learning, MDL & Information Bottleneck

Theory🔗 93 papers▶ Demo

Efficient Attention

Efficient Attention at Scale: KV Cache, GQA & FlashAttention

Efficiency⭐ 113 papers▶ Demo

RoPE

Rotary Position Embeddings (RoPE)

Representations🔗 33 papers▶ Demo

Speculative Decoding

Speculative Decoding: Lossless Multi-Token Generation

Efficiency🔗 63 papers▶ Demo

LLM Serving

LLM Serving at Scale: Prefill, Decode & Continuous Batching

Efficiency🔗 73 papers▶ Demo

MoE

Sparse Mixture of Experts: Routing, Load Balancing & Expert Parallelism

Efficiency🔗 83 papers▶ Demo

MoE Serving

MoE Serving & Scheduling: Token Dispatch, All-to-All, Disaggregated Parallelism

Efficiency🔗 43 papers▶ Demo

DPO

Direct Preference Optimization: RL-Free Alignment from Human Preferences

Scaling & Alignment🔗 53 papers▶ Demo

KTO

KTO: Alignment from Binary Feedback via Human-Aware Losses

Scaling & Alignment🔗 33 papers▶ Demo

Reward Hacking

Reward Hacking & Overoptimization: Goodhart's Law in Preference Optimization

Scaling & Alignment🔗 33 papers▶ Demo

Sparse Autoencoders

Sparse Autoencoders at Scale: Feature Dictionaries for Mechanistic Interpretability

Representations🔗 63 papers▶ Demo

Circuit Discovery

Automated Circuit Discovery: Patching, Attribution & Decomposition at Scale

Representations🔗 43 papers▶ Demo

Activation Steering

Activation Steering: Feature-Guided Interventions for Inference-Time Control

Representations🔗 43 papers▶ Demo

Long Context

Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimization

Efficiency⭐ 103 papers▶ Demo

SSMs & Hybrids

State Space Models & Hybrid Architectures: Mamba-2, Jamba, Griffin

Core Training🔗 53 papers▶ Demo

Multimodal VLP

Multimodal Foundations: Vision Encoders, Contrastive Learning & Cross-Attention Fusion

Representations🔗 53 papers▶ Demo

Tokens

Tokenization & Vocabulary Design

Representations🔗 43 papers▶ Demo

Decoding

Decoding & Sampling: Temperature, Top-p & Inference-Time Control

Core Training🔗 53 papers▶ Demo

Backprop

Backpropagation & Automatic Differentiation

Optimization🔗 42 papers

Score Matching

Score Matching & Score-Based Generative Models

Generative Models🔗 52 papers

ICL

In-Context Learning: Learning Without Weight Updates

Representations🔗 52 papers

OT/Wasserstein

Optimal Transport & Wasserstein Distance

Theory🔗 32 papers

Flows

Normalizing Flows: Exact Likelihood via Invertible Transforms

Generative Models🔗 32 papers

PPO

PPO: Proximal Policy Optimization

Scaling & Alignment🔗 11 paper

Residuals

Residual Connections & Skip Connections

Core Training🔗 21 paper

CFG

Classifier-Free Guidance in Diffusion

Generative Models🔗 21 paper

RAG

Retrieval-Augmented Generation (RAG)

Representations🔗 31 paper

Adversarial

Adversarial Examples & Robustness

Theory🔗 31 paper

Grokking

Grokking: Delayed Generalization

Theory🔗 31 paper

Logit Lens

Logit Lens: Probing Intermediate Representations

Representations🔗 41 paper

LR Schedules

Learning Rate Schedules: Warmup, Decay & Cycling

Optimization🔗 21 paper

Init

Weight Initialization: Xavier, He & µP

Optimization🔗 32 papers

Contrastive

Contrastive Learning & InfoNCE

Representations🔗 22 papers

Distributed

Distributed Training: Data, Tensor & Pipeline Parallelism

Efficiency🔗 32 papers

Beam Search

Beam Search & Structured Decoding

Core Training🔗 21 paper

Dropout

Dropout: Stochastic Regularization

Optimization🔗 21 paper

EBMs

Energy-Based Models & Score Functions

Generative Models🔗 21 paper

LayerNorm

Layer Normalization & RMSNorm

Core Training🔗 22 papers

Fisher Info

Fisher Information & Information Geometry

Theory🔗 42 papers

Natural Grad

Natural Gradient & Riemannian Optimization

Optimization🔗 32 papers

SGD+Momentum

SGD & Momentum: The Workhorses of Optimization

Optimization🔗 22 papers

AdamW

Weight Decay & AdamW: Decoupled Regularization

Optimization🔗 22 papers

Grad Clip

Gradient Clipping & Explosion Prevention

Optimization🔗 22 papers

Label Smooth

Label Smoothing & Soft Targets

Optimization🔗 22 papers

BatchNorm

Batch Normalization

Core Training🔗 12 papers

Distillation

Knowledge Distillation: Learning from Teachers

Efficiency🔗 62 papers

Quantization

Quantization: Compressing Models to Integers

Efficiency🔗 22 papers

Pruning

Pruning: Removing Unnecessary Weights

Efficiency🔗 22 papers

SSL

Self-Supervised Learning: Labels from Structure

Representations🔗 32 papers

Calibration

Calibration & Temperature Scaling

Theory🔗 32 papers

SwiGLU

SwiGLU & Gated Activations

Core Training🔗 12 papers▶ Demo

FlashAttn

FlashAttention: IO-Aware Attention

Efficiency🔗 22 papers

Constitutional

Constitutional AI: Principles-Based Alignment

Scaling & Alignment🔗 62 papers

Bregman

Bregman Divergence & Mirror Descent

Theory🔗 22 papers

RKHS

Reproducing Kernel Hilbert Spaces

Theory🔗 22 papers

TDA

Persistent Homology & Topological Data Analysis

Theory🔗 22 papers

Lie Groups

Lie Groups & Equivariant Networks

Theory🔗 22 papers

Test-Time

Test-Time Compute & Inference Scaling

Scaling & Alignment🔗 52 papers

CoT

Chain-of-Thought Prompting

Scaling & Alignment🔗 42 papers

World Models

World Models & Model-Based RL

Theory🔗 22 papers

Synth Data

Synthetic Data & Self-Improvement

Scaling & Alignment🔗 22 papers

Consistency

Consistency Models: One-Step Diffusion

Generative Models🔗 22 papers

Checkpointing

Activation Checkpointing & Memory Efficiency

Efficiency🔗 22 papers

GQA

Grouped Query Attention (GQA)

Efficiency🔗 21 paper▶ Demo

PRMs

Process Reward Models

Scaling & Alignment🔗 21 paper

RLAIF

RLAIF: AI Feedback

Scaling & Alignment🔗 21 paper

Flow Match

Flow Matching & Rectified Flows

Generative Models🔗 21 paper

Instruct

Instruction Tuning

Scaling & Alignment🔗 21 paper

Deliberative

Deliberative Alignment

Scaling & Alignment🔗 21 paper

Debate

AI Safety via Debate

Scaling & Alignment🔗 11 paper

IDA

Iterated Amplification

Scaling & Alignment🔗 31 paper

Weak→Strong

Weak-to-Strong Generalization

Scaling & Alignment🔗 21 paper

Auto RedTeam

Automated Red Teaming

Scaling & Alignment🔗 21 paper

Mesa-Opt

Mesa-Optimization & Inner Alignment

Scaling & Alignment🔗 21 paper

Sleepers

Sleeper Agents & Alignment Faking

Scaling & Alignment🔗 21 paper

LLM-as-Judge

Model-Graded Evaluations

Scaling & Alignment🔗 21 paper

Elicitation

Capability Elicitation & ELK

Scaling & Alignment🔗 21 paper

Sandwich

Sandwiching Evaluations

Scaling & Alignment🔗 21 paper

MoD

Mixture-of-Depths

Efficiency🔗 21 paper

MCTS-LLM

Tree Search over Thoughts

Scaling & Alignment🔗 21 paper

VideoWM

Video World Models

Generative Models🔗 21 paper

Self-Improve

Self-Improvement & Distillation Loops

Scaling & Alignment🔗 21 paper

Collapse

Model Collapse & Synthetic Data

Theory🔗 21 paper

InfCtx

Infinite Context Architectures

Efficiency🔗 21 paper

Why These 100 Concepts?

Complete Coverage

Together, these concepts explain the core mechanisms behind language models, diffusion models, and multimodal systems.

Missing Intuition

Each concept includes what's still poorly explained in textbooks and papers - the intuition gaps we aim to fill.

Connected Knowledge

See how concepts build on each other. Understand prerequisites and what each idea unlocks.