Skip to learning content
Learning StudioCourses and practice
Experiences

Architecture · Causal self-attention

Transformers

Self-attention builds token representations by letting tokens interact directly based on their content, while a causal language model masks future token positions.

Further reading
Token blocks sit beneath a triangular backward-visibility canopy, split into parallel attention streams, and recombine through stacked residual blocks.
Causal attention mixes only the pastEach position can gather useful earlier information in parallel, while the causal mask keeps future tokens out of view.
Evidence & limits
What the further reading establishes
An attention-based architecture can beat recurrent translation systems while taking better advantage of parallel training.
What this lab runs
You'll run the exact causal masking and scaled dot-product mixing steps on the given token representations.
What it does not prove
This lesson runs an untrained attention block. It doesn't recreate WMT training or suggest that random attention weights explain language.

Dataset

Causal Sequence Set

Source
Course-authored synthetic examples
License
Not separately licensed
Size
1 fixed six-token sequence

Further reading

Lesson progressRestoring progress…

  1. CodeRestoring
  2. ExperimentRestoring
  3. CheckRestoring

Summary

Keep track of the shapes. Start with n token representations X ∈ ℝⁿˣᵈmodel. Learned projections produce Q and K ∈ ℝⁿˣᵈk and V ∈ ℝⁿˣᵈv. That means QKᵀ has one compatibility score for every query row and key column, giving it shape n × n.

Scale the match scores. A query asks what this position needs. A key says what a position can match, and a value carries the information that may get mixed in. Divide each query-key dot product by √dₖ so the scores don't grow with the projection width and push softmax into saturated probabilities with tiny gradients.

Mask first, then normalize. For query row i, keep key columns j ≤ i and set every future score j > i to −Infinity. Then apply softmax across that row. Future positions get exactly zero probability, and the probabilities that remain add up to one and weight the rows of V.

One causal self-attention headHere's a worked three-token pass. Learned projections normally produce Q, K, and V. Scale QKᵀ by √dₖ, mask future columns before running softmax on each row, and then mix V. The reference experiment uses identity projections, so the given token-position representations act as Q, K, and V.
  1. 1
    ProjectXW → Q,K[3×d_k] · V[3×d_v]
  2. 2
    ScoreQKᵀ[3×3] / √d_k
  3. 3
    Normalizemask j>i to −∞ → softmax per row
  4. 4
    Mix valuesP[3×3]V[3×d_v] → C[3×d_v]
Rows are queries · columns are keysP = softmax(mask(QKᵀ / √d_k))
query ↓ / key →thereceiverdecoded‖context‖
the1.00001.00
receiver0.200.8000.82
decoded0.200.330.460.60

Before softmax every cell above the diagonal is −Infinity. In this small example the values use unit basis rows, so each probability row, rounded to two decimals, is also its context vector and gives the norm shown here.

The full decoder blockattention output → projection → residual + norm → MLP → residual + norm

Attention isn't the whole block. The context C = softmax(mask(QKᵀ/√dₖ))V is only the attention sublayer. A working decoder block also needs output and multi-head projections, residual paths, normalization, a position-wise MLP, and another residual path. Stacked blocks operate on representations that include both token and position information. The exercise below builds only the non-affine normalization core. Full affine layer normalization also applies a learned gain gamma and bias beta to each feature; this exercise leaves those two parameters out on purpose.

Knowledge check

When do you apply the causal mask to the attention scores?

Implementation

models/causal-transformer.py
0 of 3 exercises verifiedOpen coding workspace →

Keep the diagonal and earlier positions, and replace only scores where column > row with -Infinity before softmax.

Signature
def causal_mask(scores):
Inputs
scores square matrix [sequence × sequence]
Returns
nested list with shape [sequence × sequence]
Rule
keep score[i,j] when j ≤ i; otherwise set it to −Infinity
Example
[[1,2],[3,4]] → [[1,−Infinity],[3,4]]
Causal mask progressive practice rounds
Restoring saved code…
Reference solution

Approach Keep the diagonal and earlier positions, and replace only scores where column > row with -Infinity before softmax.

import numpy as np def causal_mask(scores):    masked = np.asarray(scores, dtype=float).copy()    if masked.ndim != 2 or masked.shape[0] != masked.shape[1]:        raise ValueError("maskCausal needs a square rank-2 tensor")    future_rows, future_columns = np.triu_indices(masked.shape[0], k=1)    masked[future_rows, future_columns] = -np.inf    return masked.tolist()

Loading saved work…

Use Python and NumPy to build the exact steps that decide which token positions can share information inside a causal attention block.

Saved results

Results created after your saved code passes its checks

The replay is course data. The validation result is tied to the code you saved and checked.

Loading…