Legacy Concept Lab

Efficiency: Quantization, Distillation, LoRA & Sparse MoE

Quantization + LoRA are standard for deploying and fine-tuning Llama-class models on modest GPUs

Concept 16 of 100EfficiencyPhase 8

#16EfficiencyEfficiency

key equation

W' = W + BA,\quad r \ll d

Phase 8: Scaling, theory & multimodalConcept 16 of 100

Why It Matters for Modern Models

Quantization + LoRA are standard for deploying and fine-tuning Llama-class models on modest GPUs
Distillation compresses large base models into "small assistants"
MoE/Switch-style sparsity powers very large Google-scale models (likely Gemini)

What Tutorials Skip

What is still poorly explained in textbooks and papers:

Geometric views of low-rank updates: LoRA as adding a small, oriented "slice" in weight space
Intuitive trade-offs in quantization: how error propagates, why some layers are more sensitive

Interactive Visualization

Core Math (Optional Deep Dive)

If you want intuition first, start with the key equation and the visualization. Come back here for the full walkthrough.

Key Equation

W' = W + BA,\quad r \ll d

Distillation: train student $q_\psi$ to match teacher $p_\theta$ :

\mathcal L = T^2\,\mathrm{KL}(p_\theta^T(\cdot\mid x)\,\|\,q_\psi^T(\cdot\mid x))

Quantization: map float weights to low-bit integers: $\tilde w = \Delta \cdot \mathrm{round}(w/\Delta)$

LoRA: re-parameterize weight matrix as: $W' = W + BA$ , where $B\in\mathbb R^{d\times r}, A\in\mathbb R^{r\times d}, r\ll d$ — only train $A,B$ , freezing $W$ .