Quality · Persistence, a11y, latency
Product Quality
A chat product should still make sense after something fails or the page reloads. You need to be able to verify every input, generation phase, announcement, recovery action, and saved record.
Evidence & limits
- What the further reading establishes
- For content that updates in order, an accessible log can keep the reading order clear and announce useful additions.
- What this lab runs
- This lesson runs 11 small executable checks, marks 5 unexecuted requirements as specifications, and lists the keyboard, screen-reader, and mobile checks a person still needs to do. The full project build adds a separate mounted behavior check.
- What it does not prove
- Automated checks help, but they don't replace testing with real browsers, keyboards, screen readers, and users.
Dataset
Product Contract Audit
- Source
- Course-authored synthetic checklist
- License
- Not separately licensed
- Size
- 11 executable pure checks · 5 specifications · 3 manual verification groups
Further reading
- ARIA23: Using role=log to identify sequential information updates
Primary · W3C Web Accessibility Initiative · WCAG 2.2
Explains how role=log identifies new items, like messages added to a conversation.
- Understanding Status Messages
Specification · W3C Web Accessibility Initiative · WCAG 2.2
Explains when assistive technology needs to announce progress, completion, and errors.
- Interaction to Next Paint
Guide · web.dev · Current
Provides a practical way to measure interaction and streaming-render responsiveness.
Lesson progressRestoring progress…
- CodeRestoring
- ExperimentRestoring
- CheckRestoring
Summary
One lifecycle you can see. Pressing Enter queues a request. Loading and prefill labels explain the wait, streaming sends updates in reasonable batches, and complete, cancelled, and error are the final states. The label on screen should come from the same phase that controls Stop and Retry.
Recovery also means focus. Stop and regenerate aren't just buttons; they move the request into a new state. Reject late events, release resources, label partial output, and put keyboard focus back on a control the user can predict.
Treat saved history as input. History loaded after a refresh is untrusted data. A versioned record should accept only the exact safe fields, size-limited final messages, and known roles and backends. Reject streaming state and any extra field that could hold a secret.
- 1Send
Enter → queued - 2Wait
loading → prefill - 3Generate
streaming → limited batches - 4Recover
cancel / retry → composer focus - 5Reload
check v1 → restore finished messages
Waiting for capacity → Loading model → Processing context → Generating → CompleteProgrammatic statestatus / polite / atomic · log / polite · batched additionsabort transport · cancel frame · reject late event · release request · focus composerSafe reloadv1 · exact keys · ≤200 terminal messages · known role/backend/status · no secretsKnow what automation can't prove. Small contract checks can verify mappings, guards, serialization, and exact policy values. A separate full-build check mounts the capstone and runs through submit, stream, stop, late-event rejection, and errors. Neither one proves focus order, what a screen reader says, how touch controls feel, or how the layout works at a real screen size. Check those by hand.
Knowledge check
Implementation
Accept only a size-limited v1 conversation record with the exact safe fields allowed for final messages.
- Signature
def valid_conversation_record(record):- Inputs
- record dict with version, id, and at most 200 terminal message dicts
- Returns
- bool validity decision
- Rule
accept only schema v1, exact allow-listed keys, bounded ids/content, terminal statuses, and JSON-serializable values- Example
a complete local message → True; the same message with apiKey → False
Reference solution
Approach Accept only a size-limited v1 conversation record with the exact safe fields allowed for final messages.
import json def valid_conversation_record(record): def is_plain_record(value): return type(value) is dict def has_exact_keys(value, required, optional=()): keys = set(value.keys()) required_keys = set(required) allowed_keys = required_keys | set(optional) return ( all(type(key) is str for key in value.keys()) and required_keys <= keys and keys <= allowed_keys ) def valid_id(value): return type(value) is str and bool(value.strip()) and len(value) <= 128 def valid_message(message): if not is_plain_record(message) or not has_exact_keys( message, ("id", "role", "backend", "content", "status"), ("attemptId", "parentUserId"), ): return False if not valid_id(message["id"]) or message["role"] not in {"user", "assistant"}: return False if message["backend"] not in {"student", "local"}: return False if type(message["content"]) is not str or len(message["content"]) > 20000: return False if message["status"] not in {"complete", "cancelled", "error"}: return False if "attemptId" in message and not valid_id(message["attemptId"]): return False if "parentUserId" in message and not valid_id(message["parentUserId"]): return False return True if not is_plain_record(record) or not has_exact_keys( record, ("version", "id", "messages"), ): return False if type(record["version"]) is not int or record["version"] != 1: return False if not valid_id(record["id"]): return False if type(record["messages"]) is not list or len(record["messages"]) > 200: return False if not all(valid_message(message) for message in record["messages"]): return False if sum(len(message["content"]) for message in record["messages"]) > 200000: return False try: return type(json.dumps(record)) is str except (TypeError, ValueError): return FalseBuild the saved-data check and user-facing phase labels in Python, then run the capstone product check.
Saved results
Results created after your saved code passes its checks
Saved results
Results created after your saved code passes its checks
The replay is course data. The validation result is tied to the code you saved and checked.
Loading…