Skip to learning content
Learning StudioCourses and practice
Experiences

Sequence models · Character prediction

Character RNNs

A recurrent network can learn a probability distribution for the next character by updating a hidden state over and over and minimizing cross-entropy through time.

Further reading
Evidence & limits
What the further reading establishes
Next-character prediction can teach a model patterns in syntax, formatting, and longer-range structure.
What this lab runs
You'll train a real vanilla RNN with truncated backpropagation and gradient clipping right in this browser tab.
What it does not prove
The provided corpus and model are intentionally tiny, so this doesn't recreate the essay's multi-layer LSTM results.

Dataset

Signal Notes

Source
Course-authored synthetic corpus
License
Not separately licensed
Size
1,610 characters · fixed repeatable sequence

Further reading

Lesson progressRestoring progress…

  1. CodeRestoring
  2. ExperimentRestoring
  3. CheckRestoring

Summary

Represent each character. Start with a small vocabulary, like {a, b, space}. Each input x_t is a one-hot vector: the current character's slot is 1, and every other slot is 0. The subscript t just means the character at the current position in the sequence.

Update the model's memory. The transition h_t = tanh(Wxh x_t + Whh h_(t-1) + b) mixes the current character with the previous hidden state. For example, after reading “th,” h_(t-1) can carry information about “t” while x_t identifies “h.” The model reuses the same Wxh, Whh, and b at every position. Only the input and state change.

Make a prediction and score it. A second projection turns h_t into one raw score, or logit, for every possible next character. Softmax turns those logits into probabilities. Cross-entropy is -log p(target): giving the real next character probability 0.8 costs about 0.22, while giving it probability 0.1 costs about 2.30.

Send credit back through time. Training unrolls several recurrent steps and sends the total loss backward through them. That's backpropagation through time, or BPTT. This lab uses a short window, called truncated BPTT, instead of the whole corpus. Since Whh gets multiplied into the gradient at every step, gradients can grow fast. Clipping puts a cap on each update before it changes the weights.

Training and generationRead from left to right. With teacher forcing, the real x_(t+1) goes into the next column and also serves as the loss target; the model doesn't feed its guess back in. During generation, x_(t+1) is sampled from the predicted distribution and fed into the next column. Both paths share Wxh, Whh, Why, and the biases.
position t − 1x_(t−1)current characterh_(t−1)new memorytanh(Wxh x_(t−1) + Whh h_(t−2) + b)logits → softmax → p(x_t)
position tx_tcurrent characterh_tnew memorytanh(Wxh x_t + Whh h_(t−1) + b)logits → softmax → p(x_(t+1))
position t + 1x_(t+1)current characterh_(t+1)new memorytanh(Wxh x_(t+1) + Whh h_t + b)logits → softmax → p(x_(t+2))
Memory flow h_(t−1) → h_t → h_(t+1)Teacher-forced training real x_(t+1) → loss target + next inputGeneration sample from p(x_(t+1)) → use it as the next inputSame at every position Wxh · Whh · Why · biases

Teacher forcing vs. sampled generation. During teacher-forced training, the real corpus character x_(t+1) is both the cross-entropy target and the next input, no matter what the model predicted. During generation, there's no known target. You sample a character from p(x_(t+1)), use that sample as the next input, update the hidden state, and repeat. The recurrent step is the same; what changes is where the next input comes from.

Knowledge check

When the model generates one character at a time, what carries information from the prefix into the next recurrent step?

Implementation

models/character-rnn.py
0 of 3 exercises verifiedOpen coding workspace →

Combine the current input with the previous hidden state.

Signature
def rnn_step(input_vector, previous, parameters):
Inputs
input_vector [vocab], previous [hidden], Wxh [hidden × vocab], Whh [hidden × hidden], bias [hidden]
Returns
list[float] with shape [hidden]
Rule
h_t = tanh(Wxh x_t + Whh h_(t−1) + b)
Example
[1, 0], [0, 0], identity Wxh → [0.762, 0]
Recurrent transition progressive practice rounds
Restoring saved code…
Reference solution

Approach Combine the current input with the previous hidden state.

import numpy as np def rnn_step(input_vector, previous, parameters):    Wxh = np.asarray(parameters["Wxh"], dtype=float)    Whh = np.asarray(parameters["Whh"], dtype=float)    bias = np.asarray(parameters["bias"], dtype=float)    input_projection = Wxh @ np.asarray(input_vector, dtype=float)    state_projection = Whh @ np.asarray(previous, dtype=float)    return np.tanh(input_projection + state_projection + bias).tolist()

Loading saved work…

Rebuild the training loop's three main operations in Python and NumPy. Start with the recurrent transition, which has to use both x_t and h_(t-1). Then build -log p(target) and symmetric clipping. Each cell runs on its own, so a mistake in one won't wipe out passing work in another.

Saved results

Recorded training replay · fixed course run, not your code

The replay is course data. The validation result is tied to the code you saved and checked.

Loading…