Continuous Function

Legacy Concept Lab

Scaling Laws & Emergent Abilities

GPT-3.5/4, Claude, Gemini, Llama were all designed with these scaling behaviors in mind

Concept 14 of 100Scaling & AlignmentPhase 8
Open New NotebookAll Foundations
#14ScalingScaling & Alignment
key equation
L(N,D,C)≈L∞+aN−α+bD−βL(N, D, C) \approx L_\infty + a N^{-\alpha} + b D^{-\beta}
Phase 8: Scaling, theory & multimodalConcept 14 of 100
Migrated:view the updated version in /domains →This /foundations page is legacy during migration.

Why It Matters for Modern Models

  • GPT-3.5/4, Claude, Gemini, Llama were all designed with these scaling behaviors in mind
  • Sora & SDXL apply similar scaling-law reasoning for image/video diffusion backbones

What Tutorials Skip

What is still poorly explained in textbooks and papers:

  • Why power-law scaling happens (statistical physics analogies, information-theoretic arguments)
  • Visual, interactive plots showing evolving task-specific performance vs scale

Interactive Visualization

Core Math (Optional Deep Dive)

If you want intuition first, start with the key equation and the visualization. Come back here for the full walkthrough.

Key Equation
L(N,D,C)≈L∞+aN−α+bD−βL(N, D, C) \approx L_\infty + a N^{-\alpha} + b D^{-\beta}

Test loss LL obeys approximate power laws:

L(N,D,C)≈L∞+aN−α+bD−βL(N, D, C) \approx L_\infty + a N^{-\alpha} + b D^{-\beta}

where NN = parameters, DD = data, CC = compute; α,β\alpha,\beta are exponents.

Chinchilla rule: for fixed compute, optimal frontier scales roughly as D∝ND \propto N — don't over-scale parameters without matching data.

Some capabilities (chain-of-thought, few-shot reasoning) appear suddenly once scale crosses a threshold — "emergent abilities."

Canonical Papers

Scaling Laws for Neural Language Models

Kaplan et al.2020arXiv
Read paper →

Training Compute-Optimal Large Language Models

Hoffmann et al.2022NeurIPS (Chinchilla)
Read paper →

Emergent Abilities of Large Language Models

Wei et al.2022TMLR
Read paper →

Connections

Prerequisites

∪Double Descent⊗Attention

Enables

⚖RLHF📝ICL🌐Distributed🧠Test-Time📉Collapse

Next Moves

Explore this concept from different angles — like a mathematician would.

Semantic Connections

⚠️ Breaks When
∪Small data→Double Descent🔤Token units shift→Tokens
≈ Analogy
⚡Budget allocation→LLM Serving🔤Compression efficiency→Tokens
🔧 Invented to Fix
🔀More params, same FLOPs→MoE
↔️ Mathematical Dual
ΘWidth ↔ Depth→NTK