Free tools Windows power users keep installed
One-click scans. No signup required.
Transformer attention lets each position use information from other positions by comparing queries with keys, turning those scores into weights, and using the weights to combine values. The query and key determine how much each value contributes; the values supply what gets combined.
What attention computes
Attention is a learned computation over vector representations—not a model consciously focusing like a person, and not a literal database lookup. A useful analogy is a search: a query expresses what a position is seeking, keys are the features it compares against, and values carry the information that can be mixed into the output. These are learned vectors, not explicit records.
As an Amazon Associate I earn from qualifying purchases.
Vaswani and colleagues describe an attention function as mapping a query and a set of key-value pairs to an output, with the query, keys, values, and output all represented as vectors. In scaled dot-product attention, the operation is:
Recommended Free Tools
Attention(Q, K, V) = softmax(QKᵀ / √dₖ)V
For one query, the computation proceeds as follows:
#1 Best Overall
- Score: Take the dot product of the query with each key. A larger score indicates a stronger match in the learned representation space.
- Scale: Divide each score by the square root of the key dimension, √dₖ.
- Normalize: Apply softmax to turn the scores into weights that sum to one.
- Combine: Multiply each value by its weight and add the results to form the output.
The score is not the retrieved content. It determines the weight; the weighted values make up the result.
Why scale the scores
Dot products can grow in magnitude as vector dimensions increase. If the scores become too large, softmax can produce very peaked weights, leaving small gradients during training. The original Transformer paper divides by √dₖ to moderate the scores before softmax and reduce that problem. This scaling changes the scores going into normalization; it does not add information to the values.
Rank #2
Self-attention and cross-attention
The distinction is where the queries, keys, and values come from. The same attention calculation can connect positions within one sequence or let one representation draw information from another.
| Type | Queries come from | Keys and values come from | What it connects |
|---|---|---|---|
| Self-attention | The sequence being updated | The same sequence representation | Positions within a sequence can exchange information. |
| Cross-attention | One representation | A separate representation | The querying representation can draw on another source. |
In the original Transformer, encoder layers use self-attention. Decoder layers also use self-attention and include encoder-decoder attention, in which decoder queries attend to encoder outputs. In either case, the query-key scores set the weights and the values provide the information being combined.
What multiple heads add
A single attention head applies one set of learned projections to form queries, keys, and values, then performs one attention operation. Multi-head attention uses several learned projection spaces and runs attention in parallel. The original paper’s motivation is that different heads can attend to information from different representation subspaces at different positions. Their outputs are combined into the layer’s result.
Multiple heads do not mean the model has a fixed set of human-like interests. They provide multiple learned ways to compute attention; what any head represents depends on the trained model.
Attention is not the whole story of sequence understanding
Attention connections alone do not tell the model which token came first. In the original Transformer, positional encodings are added to embeddings so that the model also has information about sequence order. The attention operation then works over representations that include positional information.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Likewise, an attention-weight map shows how a particular computation distributes weights over values; by itself it is not a complete or necessarily faithful explanation of a model’s reasoning. The output depends on the learned projections, the values, other layers, and the rest of the model’s computation. Treat weights as part of the mechanism, not a standalone account of why a model produced an answer.
Best Value
Scaled dot-product versus additive attention
| Scoring method | How it computes a score | Use in the original Transformer |
|---|---|---|
| Scaled dot-product | Dot product of query and key, divided by √dₖ | Selected for the Transformer’s attention mechanism. |
| Additive | A learned feed-forward function scores the query-key pair. | Discussed as an alternative scoring approach. |
Both methods turn compatibility between queries and keys into scores that can be normalized and used to weight values. They differ in how they calculate those scores.
What the original result does—and does not—show
The 2017 paper reported 28.4 BLEU for the Transformer on the WMT 2014 English-to-German translation benchmark, more than 2 BLEU above the existing best results it compared against, including ensembles. This is a result from that paper on that historical benchmark, not a current leaderboard figure or a general measure of attention quality.
Sources: Google Research, “Attention Is All You Need” (2017); NeurIPS paper (2017); Harvard NLP, “The Annotated Transformer” (2018).
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
