Self-attention is an operation that lets each position in a sequence rebuild its own representation as a weighted mix of the positions it is allowed to see. Each position scores every other visible position for relevance, turns those scores into weights that sum to 1, and adds up the corresponding value vectors. The result is a new vector for each token that now carries information from its context.
The problem self-attention solves
A sentence is a sequence of vectors, one per token. A useful representation of the word “it” in “the animal did not cross the street because it was tired” depends on other words in the same sentence. Earlier sequence models moved information along the sequence one step at a time. Self-attention instead lets every position read from every other position in a single operation. The original Transformer paper, Attention Is All You Need (Vaswani et al., NeurIPS 2017), built its entire encoder and decoder on this mechanism, dispensing with recurrence and convolutions.
As an Amazon Associate I earn from qualifying purchases.
Start with one token in a short sequence
Take a three-token sequence such as “the cat sat.” Each token already has a hidden vector, a list of numbers produced by the layer below. Self-attention computes one new vector for each token. To understand the mechanism, follow a single token, the focus token, through the calculation. The same steps repeat for every token, and in practice they are done together with batched matrix operations.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Queries, keys, and values come from the same sequence
Self-attention creates three vectors for every token. They all start from the same input sequence, which is why the operation is called “self” attention. The difference is in three learned weight matrices, one for each role. Each token’s hidden vector x is multiplied by these matrices:
#1 Best Overall
- Q = X WQ (queries)
- K = X WK (keys)
- V = X WV (values)
Here X stacks the token vectors row by row. Each row of Q, K, and V corresponds to one token.
Query
The query is the vector a position uses to search the sequence. A useful working analogy is “what this position is looking for.” The analogy describes the role the model learns to play, not a label a person assigned. The weights WQ are learned during training, so the model decides what kind of information a query should ask for.
Key
The key is the vector each position offers for matching against queries. The analogy is “what this position advertises about itself.” A query and a key that match well produce a large score. Like queries, keys are learned projections; nothing in the mechanism hard-codes what a key represents.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Value
The value is the content a position contributes once it has been selected. Scores decide how much attention a position receives, and values supply what is actually passed along. Values and keys are separate projections, which lets the model separate “how to be found” from “what to hand over.”
Rank #2
- Use scikit-learn to track an example ML project end to end
- Explore several models, including support vector machines, decision trees, random forests, and ensemble methods
- Exploit unsupervised learning techniques such as dimensionality reduction, clustering, and anomaly detection
- Dive into neural net architectures, including convolutional nets, recurrent nets, generative adversarial networks, autoencoders, diffusion models, and transformers
- Use TensorFlow and Keras to build and train neural nets for computer vision, natural language processing, generative models, and deep reinforcement learning
Scores, scaling, softmax, and the weighted sum
For the focus token, the calculation runs in four steps:
- Score. Take the dot product of the focus token’s query with the key of every visible position. A larger dot product means a stronger match.
- Scale. Divide each score by the square root of the key width, √dk. This keeps the scores in a range where softmax still produces useful gradients (explained below).
- Softmax. Apply softmax across the scores for the visible positions. Each score is exponentiated and divided by the total, so the weights are positive and sum to 1.
- Weighted sum. Multiply each weight by the corresponding value vector and add the results. The output is the focus token’s new, context-mixed vector.
Repeating these steps for every token gives one output vector per token, with the same shape as the input sequence.
A worked example with small numbers
The numbers below are invented for illustration. They are not taken from a trained model. Let the focus token have query q = [1, 0], and let the three tokens have these keys and values, with key width dk = 2:
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →- Token 1: key [1, 0], value [2, 0]
- Token 2: key [0, 1], value [0, 4]
- Token 3: key [1, 1], value [1, 1]
Dividing by √2 ≈ 1.414 and applying softmax gives the following:
Rank #3
| Key token | Dot product q·k | Scaled score (÷ √2) | Softmax weight |
|---|---|---|---|
| Token 1 | 1 | 0.71 | 0.40 |
| Token 2 | 0 | 0.00 | 0.20 |
| Token 3 | 1 | 0.71 | 0.40 |
The weighted sum is 0.40 × [2, 0] + 0.20 × [0, 4] + 0.40 × [1, 1]. Using unrounded weights, the output is approximately [1.20, 1.19]. Notice that the output is a blend of all three value vectors, not a copy of any one of them, and that the weight on token 2 is smaller because its score is lowest.
The same example with a causal mask, covered in a later section, hides token 3 from the focus token. Only tokens 1 and 2 remain. Their scores are 0.71 and 0.00, giving weights of about 0.67 and 0.33. The output becomes approximately [1.34, 1.32], because the mixture now draws only from the visible positions.
The full equation
The compact form of the whole operation is:
Attention(Q, K, V) = softmax(QKᵀ / √dk) V
Shapes make the equation easier to follow. If the sequence has n tokens, Q and K are n × dk, and V is n × dv. QKᵀ is n × n, with one score for every query-key pair. The softmax runs along each row, so each query has its own distribution over the keys it may see. Multiplying that n × n weight matrix by V, which is n × dv, produces an n × dv output: one mixed vector per token.
Free tools Windows power users keep installed
One-click scans. No signup required.
Why the score is divided by √dk
The original paper gives the reason directly. When dk is large, dot products tend to grow in magnitude, and large inputs push softmax into regions where its output is nearly one-hot and its gradients are very small. Dividing by √dk counteracts this growth. The scale is a property of the scaled dot-product formulation in the paper, so it is part of the original method rather than an optional refinement.
Rank #4
Multi-head attention
A single attention operation produces one set of weights per query. The original Transformer runs several attention operations in parallel. Each head has its own learned projections WQ, WK, and WV, so each head works in its own projected subspace. The head outputs are concatenated and multiplied by another learned matrix.
The benefit is that the model can form several different weight patterns for the same tokens at the same layer. One head may end up relating words that refer to each other, while another tracks nearby words. These roles are learned, and they are not guaranteed to be clean or human-readable, so it is safer to describe heads as parallel learned views of the same sequence.
Position information
Self-attention on its own has no notion of order. If you shuffle the input tokens, the same weighted mixture is produced for each token, just attached to different positions. Something must tell the model where each token sits.
The original Transformer adds positional encodings to the token embeddings before the first layer. It uses fixed sinusoidal functions of position, with sine and cosine terms at different frequencies, so each position receives a distinct pattern. Many later systems use learned position embeddings or relative position schemes instead. The sinusoidal version is the one described in the original paper, and it should not be assumed to be the choice of every current model.
Best Value
Causal masks and the three attention types
In a language model that predicts the next token, a position must not see the tokens that come after it, or training would leak the answer. The decoder handles this with a causal mask. Before softmax, the scores for future positions are set to negative infinity. The exponential of negative infinity is zero, so those positions receive exactly zero weight after the softmax.
Three attention types appear in the original Transformer. They differ in where queries, keys, and values come from and which positions are visible:
| Attention type | Where queries come from | Where keys and values come from | Visible positions |
|---|---|---|---|
| Encoder self-attention | Same sequence | Same sequence | All positions, in both directions |
| Decoder masked self-attention | Same sequence (output so far) | Same sequence (output so far) | Current and earlier positions only |
| Encoder-decoder attention (cross-attention) | Decoder sequence | Encoder output sequence | All encoder positions |
Only the first and second rows are self-attention in the strict sense, because all three projections start from one sequence. The third row is cross-attention, and it is the only one where the queries and the keys and values come from different sequences.
Where attention sits in a Transformer block
The attention formula is one sublayer inside a larger block. In the original design, each block applies multi-head self-attention, then adds the input back through a residual connection and normalizes the result. A position-wise feed-forward network follows, with its own residual connection and normalization. Stacking these blocks produces the full model. Understanding the formula explains the mixing step, but not the rest of what makes a Transformer work.
Common misconceptions
- Attention weights are the values. The weights come from query-key scores. They are used to mix the value vectors.
- Q, K, and V are three different tokens. They are learned projections of the same token representations.
- A high weight shows that a token is important or explains the output. A weight is the mixing coefficient in that layer and head. It does not establish semantic importance or a causal explanation of the prediction without separate evidence.
- Self-attention always sees the whole sequence. A mask can restrict which positions are visible, as in causal decoders.
- Attention is the whole Transformer. It is one sublayer in a block that also includes residual connections, normalization, and feed-forward layers.
Source attribution for the original results
The original paper reported machine-translation results on WMT 2014 English-to-French. Two published figures for its single model differ, and the difference comes from the source rather than from a calculation error. The Google Research publication page lists 41.0 BLEU, along with a training run of 3.5 days on eight GPUs. The arXiv abstract reports 41.8 BLEU for the same task. Both figures describe a 2017 result, not a current benchmark, and they are not interchangeable.
The paper’s abstract also states the central idea in the authors’ words: “We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.” Readers who want the derivation should go to the paper itself, and to Harvard NLP’s The Annotated Transformer for a line-by-line implementation. Purdue Mathematics’ “Attention from Scratch” notebook gives a stepwise conceptual treatment that complements the walkthrough above.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Recommended Free Tools
