LLM Systems

LLM Serving at Scale: Prefill, Decode & Continuous Batching

A systems mental model for LLM inference: prefill vs decode, TTFT vs TPOT, batching/scheduling, and why KV cache memory dominates.

status: publishedimportance: importantdifficulty 4/5math: undergraduateread: 22mlive demo

Back to LLM Systems Next: Speculative Decoding: Lossless Multi-Token Generation

Editorial systems illustration of prefill, KV cache shelves, continuous batching lanes, and decode token streams.

Concept Structure

LLM Serving at Scale: Prefill, Decode & Continuous Batching

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

2prerequisites

3next concepts

3related links

Learner Contract

What this page should let you do.

You are here becauseA systems mental model for LLM inference: prefill vs decode, TTFT vs TPOT, batching/scheduling, and why KV cache memory dominates.

This LLM Systems concept is the current object: keep the same invariant visible across Intuition, Math, Code, Interactive Demo.

Before thisScaled Dot-Product Attention & Transformer Layers

2 prerequisites listed; refresh them before leaning on the math or code.

Scaled Dot-Product Attention & Transformer Layers Efficient Attention at Scale: KV Cache, GQA & FlashAttention

By the end4/4 sections ready | code witness expected | live demo

Explain the mechanism, trace the main notation, and test one prediction in the live demo.

Do this firstIntuition

Read the intuition before the notation; the math should name a mechanism you already felt.

Then go nextSpeculative Decoding: Lossless Multi-Token Generation

Follow this edge after making one prediction here; the next page should reuse the result, not restart the route.

Test the linkManipulate one control and predict the visible change.Then continue to Speculative Decoding: Lossless Multi-Token Generation

Claim/source review status

Substantive review recorded

1/1 claims have bounded review metadata; still check caveats and source scope.Metadata-derived; review may be AI-assisted. Not a human certification.

Claims1/1 reviewed

Sources2 cited

Codeattached

Demolive

Reviewed2026-05-07

Updatedpage 2026-06-30

First Route Readiness

First route: attention to serving

A new learner should see one concrete path: map or search a claim, inspect the route graph, open Efficient Attention, predict the KV-cache lever, review the claim boundary, and leave a reproduction or repair note attached to the exact object.

QuestionHow does attention become a serving bottleneck?

Next repairEfficient Attention

Lab stateKV prediction lab live

Efficient Attention conceptRead the concept as the route repair, then inspect claim checks.

Use the demo prediction checkpoint and claim review before moving on.

Next: Efficient Attention concept Open Efficient Attention KV prediction checkpoint

01Homenext action 02Paper Mappernext action 03Knowledge Graphnext action 04Searchnext action 05Attention To Servingnext action +1 more

Scope labels requiredPreview/local where applicable; claim review before comparison; reproduction notes stay object-attached.

No benchmark result, hosted compute, automatic expert review, or live runtime-performance claim is made by this v0.

Object flow

4/4 sections readyAsk about this Research room

ConceptLLM Serving at Scale: Prefill, Decode & Continuous BatchingLLM Systems EquationLLM Serving at Scale: Prefill, Decode & Continuous Batching equation 1Exact equation object CodeLLM Serving at Scale: Prefill, Decode & Continuous Batching code witn...Exact code witness DemoLLM Serving at Scale: Prefill, Decode & Continuous Batching interacti...Visualization object ClaimLLM serving is a multi-iteration scheduling and memory-management pro...Exact claim check SourceOrca: A Distributed Serving System for Transformer-Based Generative M...Exact source object

ConceptLLM Serving at Scale: Prefill, Decode & Continuous BatchingLLM Systems

2 sources attachedLocal snapshot ready

concept:llm-systems/llm-serving

Codewitness nearby Predictbefore reveal Roomobject handoff

01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

Serving an LLM is not "run the model once." It's running the model for many users at once, while trying to keep latency low and the GPU busy.

Two phases dominate everything:

Prefill: process the prompt in parallel (big matrix multiplies, lots of compute).
Decode: generate one token at a time (small matmuls, but heavy KV cache reads).

This is why a system can feel fast on short prompts but collapse on long context: decode becomes memory-bandwidth bound, and the KV cache becomes the main resource you schedule around.

The practical goal is not raw throughput. It's goodput: how many requests you can serve while meeting latency SLOs.

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt

Equation 1 $\text{Latency} \approx \underbrace{\text{TTFT}}_{\text{time to first token}} + (T_{out}-1)\cdot\underbrace{\text{TPOT}}_{\text{time per output token}}.$ Equation 2 $\mathrm{Mem}_{KV} \approx B\cdot L\cdot T\cdot H_{kv}\cdot d_{head}\cdot 2 \cdot \mathrm{bytes}.$

Latency decomposition (a serving mental model)

Let $T_{out}$ be the number of generated tokens. A simple but useful decomposition is:

\text{Latency} \approx \underbrace{\text{TTFT}}_{\text{time to first token}} + (T_{out}-1)\cdot\underbrace{\text{TPOT}}_{\text{time per output token}}.

In this serving mental model, TTFT often tracks prefill.
TPOT often tracks decode (and KV cache reads).

KV cache memory scaling (why long prompts hurt)

Across a batch of $B$ sequences and $L$ layers, storing keys and values for $T$ tokens costs roughly:

\mathrm{Mem}_{KV} \approx B\cdot L\cdot T\cdot H_{kv}\cdot d_{head}\cdot 2 \cdot \mathrm{bytes}.

The factor of 2 is for storing both $K$ and $V$ .

Goodput (a serving SLO mental model)

If you have SLO thresholds $S_{TTFT}, S_{TPOT}$ , a common objective is:

\text{Goodput} = \text{Throughput} \times \Pr\!\left(\text{TTFT} \le S_{TTFT} \wedge \text{TPOT} \le S_{TPOT}\right).

This captures the reality that "fast on average" is not good enough if tail latency violates SLOs.

Paging / fragmentation waste

If KV memory is allocated in blocks/pages of size $P$ (tokens per block), then for a sequence length $T$ :

\text{allocated}(T) = \left\lceil \frac{T}{P} \right\rceil P,\qquad \text{waste}(T) = \text{allocated}(T) - T.

Block-based allocators (PagedAttention-style) make growth predictable and reduce fragmentation under continuous batching.

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt

Code witness 1import numpy as np def kv_gb(T, layers, h_kv, d_head, batch=1, bytes_per_elem=2): elems = bat...python

import numpy as np

def kv_gb(T, layers, h_kv, d_head, batch=1, bytes_per_elem=2):
    elems = batch * layers * T * h_kv * d_head * 2  # K and V
    return elems * bytes_per_elem / 1e9

def waste_tokens(T, P):
    return int(np.ceil(T / P) * P - T)

L, Hkv, dh, B = 80, 8, 128, 16  # example: 80 layers, GQA with 8 KV heads, fp16
for T in [2048, 8192, 32768, 131072]:
    print("T=", T, "KV~", round(kv_gb(T, L, Hkv, dh, batch=B), 2), "GB")

P = 256  # tokens per page/block
for T in [2000, 8192, 20000]:
    print("T=", T, "waste_tokens=", waste_tokens(T, P))

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Use the demo to explore TTFT vs TPOT tradeoffs, how batching affects goodput, and how repeated KV cache reads shape decode latency in this toy lab.

Live Concept Demo

Explore LLM Serving at Scale: Prefill, Decode & Continuous Batching

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 4/5undergraduatecode-aligned

Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what LLM Serving at Scale: Prefill, Decode & Continuous Batching should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

A systems mental model for LLM inference: prefill vs decode, TTFT vs TPOT, batching/scheduling, and why KV cache memory dominates.

Prediction open01 / Intuition

Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change LLM Serving at Scale: Prefill, Decode & Continuous Batching should make visible.

Visual Inquiry

Make the image answer a mathematical question

A systems mental model for LLM inference: prefill vs decode, TTFT vs TPOT, batching/scheduling, and why KV cache memory dominates.

4/4 stages readyLive demo connected

Visual cueWhich visible object should carry the first intuition?

Inspection depth2/4

Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make LLM Serving at Scale: Prefill, Decode & Continuous Batching easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

paper · 2022Orca: A Distributed Serving System for Transformer-Based Generative ModelsYu et al.

Primary serving-scheduler source. Orca frames generative transformer inference as multi-iteration autoregressive serving, where each iteration generates one output token, and proposes iteration-level scheduling plus selective batching.

Open source

paper · 2023Efficient Memory Management for Large Language Model Serving with PagedAttentionKwon et al.

Primary KV-cache serving source. PagedAttention frames high-throughput LLM serving as constrained by huge dynamically growing KV caches, fragmentation, batch-size limits, and paged block allocation.

Open source

Claim Review

A systems mental model for LLM inference: prefill vs decode, TTFT vs TPOT, batching/scheduling, and why KV cache memory dominates.

Status1 substantive review recorded

Claims without a substantive review badge still need exact source-support review.

Sources2 references

yu-2022-orca, kwon-2023-pagedattention

Witnesses4 local objects

Use equation, code, and demo objects to check whether the source support is operational.

Substantively reviewedLLM serving is a multi-iteration scheduling and memory-management problem: autoregressive decode advances one token at a time, batching must adapt between iterations, and KV cache memory can limit throughput.Claim metadata: source checked

Orca supports the scheduling part: generative Transformer serving is multi-iteration autoregressive inference with one output token per iteration, and fixed request-level batches motivate iteration-level scheduling plus selective batching. PagedAttention supports the memory/throughput part: large dynamic KV caches and fragmentation can limit batch size and throughput. Local math/code/demo are toy witnesses.

Sources: Orca: A Distributed Serving System for Transformer-Based Generative Models, Efficient Memory Management for Large Language Model Serving with PagedAttentionChecks the serving mechanism only. TTFT/TPOT, goodput, KV formula, code, and demo are toy witnesses, not source-derived formulas or production benchmarks. Does not certify vendor latency, scheduler optimality, hardware bottlenecks, throughput gains, or production guarantees.A bounded review summary is present; still check caveats and exact source scope.

Checked Orca and PagedAttention: Orca supports autoregressive generative Transformer serving as multi-iteration inference where each iteration emits one token, motivating iteration-level scheduling and selective batching because fixed request-level batches waste capacity. PagedAttention supports KV cache memory as large, dynamic, fragmentation-prone state that can limit batch size and throughput. Local latency/KV math, code, and demo are toy serving witnesses only.

Reviewer: codex+oracle; reviewed 2026-05-07

source-span-yu-2022-orca source-span-kwon-2023-pagedattention math-object-1 math-object-2 code-witness-1 interactive-demo

Source support candidates

paper 2022Orca: A Distributed Serving System for Transformer-Based Generative Models

paper 2023Efficient Memory Management for Large Language Model Serving with PagedAttention

Primary KV-cache serving source. PagedAttention frames high-throughput LLM serving as constrained by huge dynamically growing KV caches, fragmentation, batch-size limits, and paged block allocation.

Mechanism witnesses

Equation 1

\text{Latency} \approx \underbrace{\text{TTFT}}_{\text{time to first token}} + (T_{out}-1)\cdot\underbrace{\text{TPOT}}_{\text{time per output token}}.

Equation 2

\mathrm{Mem}_{KV} \approx B\cdot L\cdot T\cdot H_{kv}\cdot d_{head}\cdot 2 \cdot \mathrm{bytes}.

Code witness 1import numpy as np def kv_gb(T, layers, h_kv, d_head, batch=1, bytes_per_elem=2): elems = bat...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

A systems mental model for LLM inference: prefill vs decode, TTFT vs TPOT, batching/scheduling, and why KV cache memory dominates.

Readiness0/3 checks ready

Predict

Before touching the demo, predict one visible change that should happen in LLM Serving at Scale: Prefill, Decode & Continuous Batching.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Your answer canvas

Local checks

Claim

A concrete answer is on the canvas.

Mechanism

The answer names why the claim should hold.

Bridge

It touches the page context or a neighboring idea.

Misconception check

Object research drawerClose

ConceptLLM Serving at Scale: Prefill, Decode & Continuous BatchingLLM Systems

Code witness comparisonLLM Serving at Scale: Prefill, Decode & Continuous Batching code witness 1elems = batch * layers * T * h_kv * d_head * 2 # K and V Prediction before revealLLM Serving at Scale: Prefill, Decode & Continuous Batching interactive demoManipulate one control and predict the visible change.

Grounded room questionWhat is the smallest example that makes LLM Serving at Scale: Prefill, Decode & Continuous Batching click without losing the math?Local snapshot ready

Research Room

Attach the question to an exact object

Pick the concept, equation, source, code witness, claim, misconception, or demo state before asking for help. The handoff stays grounded to that object.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptLLM Systems

LLM Serving at Scale: Prefill, Decode & Continuous Batching

Anchored question

What is the smallest example that makes LLM Serving at Scale: Prefill, Decode & Continuous Batching click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action

Local action draft

This draft stays locally in this browser for concept:llm-systems/llm-serving.

Draft noteNext action

No local draft saved.

Evidence to inspect

Source ids to inspect: yu-2022-orca, kwon-2023-pagedattention
Definition, prerequisite, and contrast concept links
The equation or code witness that makes the concept operational
One demo state that shows the invariant instead of a slogan

What would resolve this

The learner can state the mechanism in their own words
The learner can name the prerequisite that would repair confusion
The learner can predict how the mechanism changes under one perturbation

Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - LLM Serving at Scale: Prefill, Decode & Continuous Batching Object key: concept:llm-systems/llm-serving Context: LLM Systems Anchor id: concept/concept-notebook/llm-systems/llm-serving Open question: What is the smallest example that makes LLM Serving at Scale: Prefill, Decode & Continuous Batching click without losing the math? Evidence to inspect: - Source ids to inspect: yu-2022-orca, kwon-2023-pagedattention - Definition, prerequisite, and contrast concept links - The equation or code witness that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

Open source object

concept/concept-notebook/llm-systems/llm-serving
concept:llm-systems/llm-serving

Learning Map

Before / Now / Try / Next

BeforeScaled Dot-Product Attention & Transformer Layers

NowIntuition → Math → Code → Demo

TryManipulate one control and predict the visible change.

NextSpeculative Decoding: Lossless Multi-Token Generation

Intuitionready
Mathready
Codeready
Interactive Demoready

Object Companion

Ask beside the selected object

A systems mental model for LLM inference: prefill vs decode, TTFT vs TPOT, batching/scheduling, and why KV cache memory dominates.

Your question

GoalComfortStyleStuck on

Context prompt

You are my AI learning companion for Continuous Function. Current context: LLM Systems concept. Learning surface: LLM Serving at Scale: Prefill, Decode & Continuous Batching. What this page says: A systems mental model for LLM inference: prefill vs decode, TTFT vs TPOT, batching/scheduling, and why KV cache memory dominates. Current section: Intuition, math, code, and interactive demo. Suggested next step: Manipulate one control and predict the visible change.. Learner goal: Understand the idea. Learner comfort level: New to this. Preferred explanation style: Visual first. Task: Explain the central idea in plain language, then restate it with the exact math objects from the page. Answer in a way that helps me learn: ask one clarifying question only if needed, use intuition before notation, and end with one thing I should try on the page.

Domain

LLM Systems

llm-systemsservingbatchingkv-cachelatency

Prerequisites

Scaled Dot-Product Attention & Transformer Layers Efficient Attention at Scale: KV Cache, GQA & FlashAttention

Leads To

Speculative Decoding: Lossless Multi-Token Generation Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimization Cost and Latency Observability

Efficiency: Quantization, Distillation, LoRA & Sparse MoE Tokenization & Vocabulary Design Evaluation Pipelines

Within this domain