Self-attention lets each token’s vector gather information from other token positions. A layer projects the sequence into queries, keys, and values; compares each query with the keys; and uses the resulting weights to combine the values. In short: queries and keys decide how much to attend, while values provide the information that gets mixed.
What queries, keys, and values mean
Before attention is calculated, each token has a vector representation. A self-attention layer applies three learned linear projections to the input vectors, usually written as:
As an Amazon Associate I earn from qualifying purchases.
Q = XWQ, K = XWK, V = XWV
Here, X is the matrix of token vectors and WQ, WK, and WV are learned parameter matrices. The terms are helpful analogies, not literal labels attached to words: a query expresses what a position can match for, a key is what another position can be matched against, and a value is the information that position can contribute.
Recommended Free Tools
How the self-attention calculation works
For one position, the layer compares its query with the keys at the positions it is allowed to consider. Each query-key dot product produces a compatibility score. A higher score means a stronger match in the model’s learned space, not an objective or necessarily human-readable measure of semantic similarity.
- Project the token vectors. Calculate Q, K, and V from the input sequence using the learned matrices.
- Compare the query with the keys. Take dot products between that query and each available key to produce scores.
- Scale the scores. Divide each score by the square root of dk, the dimension of the keys and queries.
- Turn scores into weights. Apply softmax across the available key positions. The resulting weights sum to one.
- Mix the values. Multiply each value vector by its corresponding weight and add the weighted vectors together.
- Repeat for each position. Each position gets its own weighted mixture. Matrix operations compute the positions together efficiently.
The result at a query position is a contextualized vector: a weighted combination of value information from the sequence. The attention scores themselves are not what gets aggregated; they are used to set the weights for aggregating values.
The equation and why it is scaled
The full operation is commonly written:
Attention(Q, K, V) = softmax(QKT / √dk)V
In their 2017 paper, “Attention Is All You Need”, Vaswani and colleagues explain that dot products can grow large as the key dimension increases. Large scores can push softmax into regions with very small gradients, making learning harder. Dividing by √dk moderates the scores before softmax.
Rank #2
What makes it self-attention—and when positions are masked
It is self-attention because Q, K, and V are all derived from the same input sequence, even though each uses a different learned projection. That lets the representation at one position incorporate information from other positions.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Which positions are available depends on the layer’s purpose. Encoder self-attention can consider positions throughout the input. In causal language-modeling, a mask prevents a position from attending to future tokens, so predictions do not use information that has not appeared yet. The original Transformer paper describes this restriction in its decoder. Self-attention does not, by itself, encode token order; the Transformer uses positional encodings to provide position information.
Rank #3
How multi-head attention extends the calculation
Multi-head attention runs several learned sets of Q, K, and V projections in parallel. Each head calculates its own weighted value mixtures. The layer concatenates the head outputs and applies an output projection to combine them. The learned heads can capture different attention patterns, but a particular head does not have a guaranteed, fixed linguistic role.
How self-attention differs from related mechanisms
- Self-attention versus cross-attention: self-attention derives queries, keys, and values from the same sequence. In cross-attention, queries come from one sequence while keys and values come from another.
- Causal versus unmasked self-attention: causal attention blocks future positions; unmasked attention can use all positions available to the layer.
- Single-head versus multi-head attention: a single head uses one set of projections; multi-head attention computes several in parallel and combines their outputs.
- Scaled dot-product versus additive attention: these are different ways to calculate compatibility scores. The equation above describes scaled dot-product attention.
What the original Transformer results do—and do not—show
The 2017 paper reported 28.4 BLEU for its Transformer base model on WMT 2014 English-to-German and 41.8 BLEU for its Transformer big model on WMT 2014 English-to-French. The authors reported training the English-to-French model for 3.5 days on eight GPUs. These are results from the paper’s specific 2017 translation experiments, not current benchmark comparisons.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
