Skip to learning content
Learning StudioCourses and practice
Experiences

Operations · Timeouts, retries, metrics

Reliability and Observability

A streaming chat request has state. When it fails, you need to classify the failure, limit retries, measure what happened, and explain it to the user without duplicating text or messing up the conversation.

Further reading
Evidence & limits
What the further reading establishes
For a distributed system people use, watch latency, traffic, errors, and saturation. Pay attention to the full spread and to what users notice, not just averages.
What this lab runs
Four fixed failure traces show the logical request id, each attempt id, when visible output makes a retry unsafe, planned timings, final state changes, rejected late events, and released resources.
What it does not prove
Local traces can't recreate a provider outage, a cross-region network split, or real production traffic patterns.

Dataset

Failure Trace

Source
Course-authored synthetic scenarios
License
Not separately licensed
Size
4 failure modes · fixed seeds

Further reading

  • Monitoring Distributed Systems

    Primary · Google SRE · 2016

    Explains why latency, traffic, errors, and saturation are the monitoring signals that matter most.

  • Alerting on SLOs

    Guide · Google SRE Workbook · 2018

    Shows how to turn error budgets and burn rates into alerts people can act on.

  • OpenTelemetry Trace specification

    Specification · OpenTelemetry · Current

    Defines spans, context, events, status, sampling, and export rules for distributed traces.

Lesson progressRestoring progress…

  1. CodeRestoring
  2. ExperimentRestoring
  3. CheckRestoring

Summary

Track the request and each try. Keep the same logical request id from start to finish, but assign a new attempt id to every retry. Put the active attempt id on stream events, logs, metrics, cancellation, and owned resources so a delayed event from an older attempt can't change the current one.

Know when an automatic retry is safe. Only retry automatically when the failure is temporary, the user hasn't seen any tokens, and you still have another attempt. In this practice API, attempt is a zero-based index and maxAttempts is the total number of tries allowed, so the check is attempt + 1 < maxAttempts.

Block events after the request ends. Only queued, loading, prefill, and streaming requests can accept matching events. Reject events for complete, error, cancelled, or unknown requests, along with events from an old attempt, before they touch conversation state.

One request across two attemptsThe logical request keeps the same id after a retry, but the attempt id changes. Once a token is visible, an automatic retry is off the table. Reject events from finished or old attempts before they can change state.
Logical requestr-201Attempt budget2 total · index 0–1ID ruleone active attempt id
  1. 1
    Attempt r-201.1queue 120 ms → transient timeout · visible 0
  2. 2
    Retry decisiontransient ∧ visible = 0 ∧ 0 + 1 < 2 → retry
  3. 3
    Attempt r-201.2queue 14 ms + prefill 69 ms → TTFT 83 ms
  4. 4
    Terminal10 deltas · decode 338 ms → complete · resources released
  5. 5
    Late-event guardr-201.1 token rejected · r-201.2 post-complete token rejected
What happened · retrytransient · visible 0 · 0 + 1 < 2retire r-201.1 → create r-201.2If a token were visible · stoptransient · visible 1keep the partial output → final error
Attempt 1 queue
120 ms
Attempt 2 queue
14 ms
Prefill
69 ms
TTFT
83 ms
Decode
338 ms
End to end
541 ms
Stale attemptevent r-201.1 ≠ active r-201.2 → rejectTerminal attemptr-201.2 status complete → reject

Measure each phase. For every request and attempt, record queue time, prefill time, time to first token, decode duration or inter-token latency, the final result, and whether resources were released. Histograms and error categories show slow tails that one average can hide. Fixed failure injection lets you exercise those paths on purpose.

Knowledge check

Under this lesson's rules, when is it safe to retry generation automatically?

Implementation

backend/generation-reliability.py
0 of 2 exercises verifiedOpen coding workspace →

Retry only temporary failures that happen before visible output, and never go past the attempt limit.

Signature
def should_retry(options):
Inputs
options {transient: bool, tokensEmitted: int, attempt: int, maxAttempts?: int}
Returns
bool retry decision
Rule
transient and tokensEmitted == 0 and attempt + 1 < maxAttempts
Example
transient, 0 visible tokens, attempt 0 of 2 → True; 3 visible tokens → False
Retry policy progressive practice rounds
Restoring saved code…
Reference solution

Approach Retry only temporary failures that happen before visible output, and never go past the attempt limit.

def should_retry(options):    transient = options["transient"]    tokens_emitted = options["tokensEmitted"]    attempt = options["attempt"]    max_attempts = options.get("maxAttempts", 2)    return transient and tokens_emitted == 0 and attempt + 1 < max_attempts

Loading saved work…

Build the retry rules and final-state guards in Python, then inject failures into a streaming request trace.

Saved results

Results created after your saved code passes its checks

The replay is course data. The validation result is tied to the code you saved and checked.

Loading…