Skip to learning content
Learning StudioCourses and practice
Experiences

Alignment · Additive attention

Additive Attention

At each output step, the decoder can build a new context vector by learning a soft alignment—how much to focus—over every encoder state.

Further reading
A decoder query shines weighted beams over source-memory cubes and gathers them into a context, contrasted with a second sequence squeezed through one fixed bottle-shaped bottleneck.
Attention builds a fresh context for each outputInstead of squeezing the whole source into one fixed summary, the decoder can place a different weight on every source memory at every step.
Evidence & limits
What the further reading establishes
Learned soft alignment eases the fixed-length bottleneck in neural translation.
What this lab runs
You'll train a real additive scorer to match output roles with the given encoder states and produce a learned heatmap.
What it does not prove
This small experiment uses labeled alignment roles instead of the paper's end-to-end translation goal.

Dataset

Date Alignment

Source
Course-authored synthetic task
License
Not separately licensed
Size
3 semantic roles · 3 fixed alignment cases · 2,000 epochs

Further reading

Lesson progressRestoring progress…

  1. CodeRestoring
  2. ExperimentRestoring
  3. CheckRestoring

Summary

The fixed-vector squeeze. Without attention, the encoder has to squeeze all n source positions into one fixed vector before decoding starts. The decoder gets that same summary for every output token. That means details needed late in a long sequence have to survive both the initial squeeze and many recurrent updates.

One decoding step. The encoder reads the source sequence and leaves one state h_i at each position. The decoder produces the target one token at a time. Right before output step t, its current state becomes the query q_t, which can look back at every encoder state. In notation, q_t has shape [d_s], and H = [h_1, ..., h_n] has shape [n, d_h]. The same scorer runs n times—once for q_t paired with each h_i—and produces one number e_(t,i) per source position.

Additive score. The score is e_(t,i) = v^T tanh(Wq q_t + Wk h_i + b). Wq and Wk project the query and key into the same attention width d_a. Then tanh combines them, and v turns the d_a values into one number. That's additive attention. A dot-product scorer uses q_t^T h_i instead, with no scoring MLP inside the comparison.

Normalize across positions. For a given t, apply softmax across all n source-position scores: alpha_(t,:) = softmax(e_(t,:)). The weights are positive and add up to 1. In the date task, the row that outputs the year should put most of its weight on the source state for 2026. A large standalone score isn't enough; it has to win against the other positions.

One output step: emit yearThe experiment runs this calculation for the year, month, and day. Read each heatmap row across the source positions. A focused row has one alpha near 1, while uniform attention stays at 0.333 for all three positions.
  1. 1
    Queryq_year [d_s]
  2. 2
    Encoder statesH = [h_day, h_month, h_year] [3 × d_h]
  3. 3
    Additive scorese = [-1.8, -0.9, 2.4] [3]
  4. 4
    Alignmentalpha = softmax(e) = [.014, .035, .951] [3]
  5. 5
    Contextc_year = .014h_day + .035h_month + .951h_year [d_h]
Additivevᵀ tanh(Wq q + Wk h_i + b)learned projections + nonlinear scoreDot productqᵀ h_idirect similarity; not the method used here

Build the context. The context for this step is c_t = sum_i alpha_(t,i) h_i, with shape [d_h]. Multiply each encoder state by its matching alignment weight, then add the results coordinate by coordinate. The decoder uses c_t for the current output. At the month and day steps, a new query creates new scores, weights, and context. Since every operation is differentiable, the translation loss can train the alignment along with the rest of the model.

Knowledge check

When one decoder query attends to a source sequence, which axis does softmax use?

Implementation

models/additive-attention.py
0 of 3 exercises verifiedOpen coding workspace →

Score one decoder query against one encoder state.

Signature
def additive_score(query, key, parameters):
Inputs
query [q], key [k], Wq [a × q], Wk [a × k], bias and v [a]
Returns
finite float compatibility score
Rule
e = vᵀ tanh(Wq q + Wk k + b)
Example
q=[1,0], k=[0,1], identity projections, v=[0.5,−0.5] → 0
Compatibility score progressive practice rounds
Restoring saved code…
Reference solution

Approach Score one decoder query against one encoder state.

import numpy as np def additive_score(query, key, parameters):    Wq = np.asarray(parameters["Wq"], dtype=float)    Wk = np.asarray(parameters["Wk"], dtype=float)    v = np.asarray(parameters["v"], dtype=float)    bias = np.asarray(parameters["bias"], dtype=float)    query_term = Wq @ np.asarray(query, dtype=float)    key_term = Wk @ np.asarray(key, dtype=float)    hidden = np.tanh(query_term + key_term + bias)    return float(v @ hidden)

Loading saved work…

Build one attention step in three separate Python/NumPy cells. First score one query-key pair with the additive MLP. Then run softmax over all source-position scores. Finally, multiply each state by its matching alpha and add the results coordinate by coordinate.

Saved results

Results created after your saved code passes its checks

The replay is course data. The validation result is tied to the code you saved and checked.

Loading…