Legacy Concept Lab

MoE Serving & Scheduling: Token Dispatch, All-to-All, Disaggregated Parallelism

MoE inference is not just "less FLOPs"—routing creates skew, and the busiest expert/GPU determines latency (straggler problem)

Concept 23 of 100EfficiencyPhase 6

Open New Notebook All Foundations

#23MoE ServingEfficiency

key equation

t_{\text{layer}} = \max_{e} \frac{n_e \cdot d_{\text{model}} \cdot d_{\text{ffn}}}{\text{FLOPS}_e}

Phase 6: Modern efficiency & inferenceConcept 23 of 100

Why It Matters for Modern Models

MoE inference is not just "less FLOPs"—routing creates skew, and the busiest expert/GPU determines latency (straggler problem)
Every MoE layer does all-to-all communication (dispatch tokens to experts, combine results)—this is memory/network bound, not compute bound
MegaScale-Infer shows 1.90× per-GPU throughput by disaggregating attention vs expert FFNs—separating workloads enables specialization
Mixtral, DeepSeek-V2/V3, DBRX all face these serving constraints—understanding MoE serving explains real production deployment decisions
After #21 (serving) and #22 (MoE routing), #23 explains what actually breaks when you combine them at scale

What Tutorials Skip

What is still poorly explained in textbooks and papers:

Sparsity buys FLOPs but sells you a scheduling problem—load becomes bursty and skewed, busiest expert dictates latency
MoE inference is two collective communications per layer (dispatch/combine)—not just matmuls, but all-to-all patterns + synchronization
Batch size is not "free" in MoE decoding—you need enough tokens per expert for GEMM efficiency, but you're constrained by KV cache + SLOs
Expert parallelism changes what scales—you're scaling placement, routing-induced traffic, and microbatching strategy, not just tensor parallelism
Disaggregation is the new architecture knob—attention and expert FFNs can be scaled/deployed differently with pipelining to keep both busy

Interactive Visualization

Core Math (Optional Deep Dive)

If you want intuition first, start with the key equation and the visualization. Come back here for the full walkthrough.

Key Equation

t_{\text{layer}} = \max_{e} \frac{n_e \cdot d_{\text{model}} \cdot d_{\text{ffn}}}{\text{FLOPS}_e}

MoE serving transforms "sparse compute" into a systems scheduling problem—routing creates skew, fragmentation, and communication overhead.

Token dispatch (all-to-all pattern):

Per MoE layer, tokens route to experts creating a permutation. For batch $T$ tokens routing to $E$ experts:

\text{Bytes}_{\text{comm}} \approx 2 \cdot T \cdot k \cdot d_{\text{model}} \cdot b

where $k$ is top-k, $d_{\text{model}}$ is hidden dimension, $b$ is bytes/element. This is dispatch + combine communication.

Straggler latency (skew problem):

With routing skew, per-expert load varies. Layer time is dominated by the busiest expert:

t_{\text{layer}} = \max_{e \in [1,E]} t_e = \max_e \left(\frac{n_e \cdot d_{\text{model}} \cdot d_{\text{ffn}}}{\text{FLOPS}_e}\right)

where $n_e$ is tokens routed to expert $e$ . Even with low average load, tail latency kills throughput.

Disaggregation tradeoff:

MegaScale-Infer separates attention from expert FFNs on different GPU pools:

\text{Utilization}_{\text{total}} = f(\text{attention-pool}, \text{expert-pool}, \text{pipeline-depth})

Trading communication for specialization and better resource allocation.

Canonical Papers

MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism

Zhu et al.2025arXiv

Read paper →

Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Scheduling

Li et al.2024arXiv

Read paper →

Mixtral of Experts

Jiang et al.2024arXiv

Read paper →

Connections

Prerequisites

⊗Attention ⚙Efficient Attention ⚡LLM Serving 🔀MoE

Next Moves

Explore this concept from different angles — like a mathematician would.

Semantic Connections

🔄 Same Technique

⏩Hide latency with parallelism→Speculative Decoding 📏Memory paging strategies→Long Context

⚠️ Breaks When

🔀Routing stragglers→MoE

🔧 Invented to Fix

🔀MoE scheduling→MoE

≈ Analogy

⚡Scheduling under skew→LLM Serving