Skip to learning content
Learning StudioCourses and practice
Experiences

Inference · Prefill and decode

Inference Runtime

LLM inference handles the prompt in a parallel prefill phase, then decodes one position at a time. The key-value cache grows with the cached sequence positions.

Further reading
Several rows of prompt cubes enter a processor together, narrow into a single-file token stream, and fill an expanding shelf of cache pages below.
Prefill is wide; decoding is one step at a timeThe prompt can be processed in parallel, but generated tokens arrive serially while the key-value cache grows with the sequence.
Evidence & limits
What the further reading establishes
Paged allocation cuts down on wasted KV-cache space and can raise serving throughput when request lengths vary.
What this lab runs
A fixed browser trace lets you inspect prefill, repeated decode, worker isolation, cache growth, and the planned timing for each phase.
What it does not prove
This doesn't recreate a GPU kernel, a network of multiple hosts, or a production memory allocator.

Dataset

Inference Trace

Source
Course-authored synthetic requests
License
Not separately licensed
Size
6 requests · fixed prompt and output lengths

Further reading

Lesson progressRestoring progress…

  1. CodeRestoring
  2. ExperimentRestoring
  3. CheckRestoring

Summary

Two phases. Prefill processes every prompt position in parallel, saves the keys and values, and returns the logits used to sample the first output token. To generate N tokens total, the runtime then does max(0, N - 1) one-position decode forwards for the rest.

How memory grows. Every processed sequence position adds one key and one value at each layer. The bytes for one request are 2 × layers × KV heads × tokens × head dimension × bytes per value. With grouped-query attention, KV heads matter here, not query heads.

Keep it off the UI thread. Run browser inference in a Web Worker so the model can't freeze React rendering. Messages give you a clear boundary between product state and model state.

Worked request r-104A 96-token prompt and a 32-token output need one prefill forward and 31 later decode forwards. The final sequence has 128 tokens.
Requestr-104Prompt96 tokensOutput32 tokensFinal length128 tokens
  1. 1
    Queue18 ms
  2. 2
    Prefill74 ms · 96 positions · 6 KV pages
  3. 3
    First tokensampled at TTFT 92 ms
  4. 4
    Decode31 forwards · tokens 2–32 · 6 → 8 pages
  5. 5
    Release8 pages returned
TTFTqueue + prefill = 18 + 74 = 92 msrequest accepted → first visible tokenITLgap between visible tokenshow quickly one request keeps decodingtokens/s21.4 generated / secondsteady decode rate
Per-request KV-cache bytes2 × layers × KV heads × cached tokens × head dimension × bytes / value2 means one key tensor + one value tensor. Under GQA, KV heads may be fewer than query heads.

Measure what the user feels. Time to first token (TTFT) runs from admission to the first token on screen, so it includes queueing and prefill. Inter-token latency (ITL) is the gap between later visible tokens. Tokens per second gives the steady decode rate. If you roll all of that into one duration, you lose the shape of the wait the user actually experiences.

Knowledge check

Which phase handles every prompt position before the first token is generated?

Implementation

systems/inference-runtime.py
0 of 2 exercises verifiedOpen coding workspace →

Keep generated tokens, model forwards, processed positions, and final sequence length straight.

Signature
def inference_phases(prompt_tokens, max_new_tokens):
Inputs
prompt_tokens int ≥ 0, max_new_tokens int
Returns
dict[str, int] of prefill, decode, processed, and final counts
Rule
generated=max(0,N); decode=max(0,generated−1); final=prompt+generated
Example
prompt 96, new 32 → 31 decode forwards and final length 128
Phase accounting progressive practice rounds
Restoring saved code…
Reference solution

Approach Keep generated tokens, model forwards, processed positions, and final sequence length straight.

def inference_phases(prompt_tokens, max_new_tokens):    generated_tokens = max(0, max_new_tokens)    decode_forwards = max(0, generated_tokens - 1)    return {        "prefillTokens": prompt_tokens,        "generatedTokens": generated_tokens,        "decodeForwards": decode_forwards,        "processedTokenPositions": prompt_tokens + decode_forwards,        "finalSequenceLength": prompt_tokens + generated_tokens,    }

Loading saved work…

Build phase accounting and KV-cache sizing in Python, then replay the planned request timeline.

Saved results

Results created after your saved code passes its checks

The replay is course data. The validation result is tied to the code you saved and checked.

Loading…