Skip to learning content
Learning StudioCourses and practice
Experiences

Embeddings · Next-token prediction

Neural Language Models

What the model learns about one word can help it predict related words, instead of treating every context as unrelated.

Further reading
Evidence & limits
What the further reading establishes
Distributed word representations help models go beyond the exact n-grams they saw during training.
What this lab runs
You'll train a small context-embedding model and compare it with a count-based baseline on held-out examples.
What it does not prove
The lab leaves out the paper's deeper hidden network, large dataset, and full vocabulary-softmax cost.

Dataset

Roles and Actions

Source
Course-authored synthetic corpus
License
Not separately licensed
Size
20 sentences · fixed example-by-example modulo split

Further reading

Lesson progressRestoring progress…

  1. CodeRestoring
  2. ExperimentRestoring
  3. CheckRestoring

Summary

How an n-gram counts. An n-gram model guesses the next word from an exact window of earlier words. A trigram estimate for “the researcher reads” depends on how often that exact three-word sequence showed up. If the model never saw “the researcher,” that count gives it no direct help—even if it saw “the analyst reads” many times.

Look up the embeddings. Give every word in the vocabulary an integer id, then use that id to pick a row from a trainable embedding table. Each row is a short vector of continuous values. For a two-word context, this lab looks up both rows and averages them coordinate by coordinate to make one context vector.

Score every possible next word. A learned projection turns the context vector into one logit—a raw score—for every possible next word. Softmax exponentiates the score differences and divides by their total. Subtracting the largest logit first doesn't change the probabilities, but it does prevent numerical overflow.

Score the real next word. Training scores the real next word with −log p(target). With 30 vocabulary items, a uniform model gives each one probability 1/30 and gets a loss of ln(30) ≈ 3.40. A validation loss of 2.53 means a perplexity of exp(2.53) ≈ 12.6. In effect, the model has cut its next-word uncertainty from 30 equally likely choices to about 12.6.

Learn both parts together. Backpropagation updates the output projection and the chosen embedding rows together. Words end up with nearby vectors only when similar coordinates help predict what follows them. The nearest-neighbor list after training shows the shape the model learned; it isn't a dictionary of meanings someone entered by hand.

Context vectors to next-word lossHere's a small numerical example with a three-word output vocabulary. The live experiment runs the same steps over 30 words and learns both the embedding table and the output projection.
Exact count“the researcher” unseen → no direct trigram estimateLearned vectorssimilar predictive use → nearby embeddings
  1. 1
    Context idsthe analyst → [4, 17]
  2. 2
    Embedding lookup[.6, −.2], [.2, .8]
  3. 3
    Mean contextc = [.4, .3]
  4. 4
    Vocabulary logitsz = [1.2, .1, −.4]
  5. 5
    Stable softmaxp = [.65, .22, .13]
  6. 6
    Target loss−log .65 = .43
Output order reads · writes · sleepsObserved target reads

A big vocabulary costs more. This lab has 30 output words, but a production vocabulary can have tens of thousands of tokens. Creating and normalizing one logit for every item makes the output layer expensive. Modern models use different context encoders, but embedding lookup, logits, stable softmax, and negative log-likelihood are still basic building blocks.

Knowledge check

If the word that actually comes next gets probability 0.8, which expression gives its cross-entropy contribution?

Implementation

models/neural-language-model.py
0 of 3 exercises verifiedOpen coding workspace →

Turn vocabulary logits into probabilities without exponent overflow.

Signature
def stable_softmax(logits):
Inputs
logits [vocab]
Returns
list[float] with shape [vocab], summing to 1
Rule
p_i = exp(z_i − max(z)) / Σ_j exp(z_j − max(z))
Example
[1001, 1000, 999] → [0.665, 0.245, 0.090]
Stable softmax progressive practice rounds
Restoring saved code…
Reference solution

Approach Turn vocabulary logits into probabilities without exponent overflow.

import numpy as np def stable_softmax(logits):    values = np.asarray(logits, dtype=float)    if values.size == 0:        return []    shifted = values - np.max(values)    weights = np.exp(shifted)    return (weights / weights.sum()).tolist()

Loading saved work…

Implement stable softmax, average the context embeddings, and compute the target loss. Run each function independently before training.

Saved results

Results created after your saved code passes its checks

The replay is course data. The validation result is tied to the code you saved and checked.

Loading…