Each output vector in a Transformer’s self-attention layer is a weighted sum of value vectors. The weights come from comparing one token’s query with every other token’s key, then normalizing those comparisons. The cost grows with the square of sequence length because standard full self-attention compares every token with every other token, creating a score grid that expands with n² as the sequence gets longer.
What gets averaged
Self-attention takes a sequence of token vectors and produces a new vector for each position. Each new vector is built by mixing together the value vectors of all positions in the sequence. “Averaged” is the right intuition, but the mixing is not a plain arithmetic mean. The coefficients differ for every query position and every input, and they are learned during training.
As an Amazon Associate I earn from qualifying purchases.
In compact form, for one attention head, the computation is usually written as:
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Attention(Q, K, V) = softmax(Q KT / √dk) V
Here Q, K, and V are matrices holding the query, key, and value vectors for each token, and dk is the key dimension. The division by √dk is a scaling step from the original paper that keeps the scores from growing too large before the softmax.
#1 Best Overall
- Use scikit-learn to track an example ML project end to end
- Explore several models, including support vector machines, decision trees, random forests, and ensemble methods
- Exploit unsupervised learning techniques such as dimensionality reduction, clustering, and anomaly detection
- Dive into neural net architectures, including convolutional nets, recurrent nets, generative adversarial networks, autoencoders, diffusion models, and transformers
- Use TensorFlow and Keras to build and train neural nets for computer vision, natural language processing, generative models, and deep reinforcement learning
Where the weights come from
The weights are not stored as a fixed table. They are computed from the input, through three learned projections. The original Transformer paper, Vaswani et al. (2017), describes the mechanism in these steps:
- Each token’s embedding is multiplied by three learned weight matrices to produce a query, a key, and a value vector.
- For a given query token, its query vector is compared against the key vector of every token in the sequence, usually with a dot product. This produces one compatibility score per token pair.
- The scores for that query are passed through a softmax, which turns them into positive weights that sum to 1.
- The output for that position is the sum of all value vectors, each multiplied by its weight.
Because the projections are learned, the model decides during training what counts as “relevant” to what. Because the scores depend on the input, the same position can attend to different tokens in different sentences. Multi-head attention runs several of these projection sets in parallel, so the model can compute several different weighted averages for each position and then combine them.
Why the cost grows with the square
The expensive part is step 2. In standard full self-attention over n tokens, each query scores against all n keys. Doing that for all n queries produces an n × n matrix of scores for each head. Doubling the sequence does not double the work; it multiplies the number of pairwise scores by four.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesRank #2
A concrete count makes the scale clear. A sequence of 1,000 tokens produces 1,000,000 pairwise scores per head in a single attention layer. A sequence of 2,000 tokens produces 4,000,000. A sequence of 8,000 tokens produces 64,000,000. The per-head score matrix is the object that drives memory use, and the number of multiply-add operations to fill it drives runtime.
NVIDIA’s Transformer Engine 2.15.0 documentation states the consequence directly: for the attention calculation it describes, runtime and memory requirements quadruple when sequence length doubles. That statement describes that documented calculation in that release. It is the clearest practical reading of the quadratic scaling, and it is why long contexts are a constant engineering problem.
What the square does not mean
The quadratic term applies to the standard full-attention calculation. It does not mean that every operation in a Transformer grows with n². The projections that create queries, keys, and values, and the feed-forward layers that follow attention, scale linearly with the number of tokens. The n² behavior comes from the pairwise attention step specifically.
It also does not mean that long sequences are impossible. It means that, without changing the computation, the attention part of a model becomes the dominant cost as the context grows. Everything that follows is a response to that one term.
Two different ways to reduce the cost
Methods for handling long sequences fall into two groups that are often mixed together. One group keeps the exact full-attention result and changes how memory is used to compute it. The other changes the attention operation itself, which can change what the output is.
Memory-optimized exact attention
NVIDIA’s documentation describes flash attention as an exact method that improves memory efficiency and data movement through tiling and recomputation. Tiling processes the attention computation in blocks that fit in fast on-chip memory, rather than writing the whole score matrix out at once. Recomputation means some intermediate values are recalculated when needed rather than stored.
Rank #4
The flash algorithm does not store the full softmax matrix for the backward pass. Instead, it saves normalization factors, which are small, and rebuilds what it needs from them. This lowers memory use. It does not remove the pairwise interactions: the method still performs full pairwise attention, so it should not be read as making the underlying computation linear.
Linear attention
Katharopoulos et al. (2020), in Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention (Proceedings of Machine Learning Research, volume 119), take a different route. They replace the softmax similarity with kernel feature maps, which lets the computation be rearranged using matrix associativity. The rearrangement avoids forming the n × n score matrix, and the paper states linear O(N) complexity in sequence length for this reformulated attention.
This is a change to the attention formulation, so the outputs are not the same as standard softmax attention. The paper reports experiments in which its Linear Transformers were up to 4000 times faster for autoregressive prediction of very long sequences. That figure comes from the paper’s own setup and should not be generalized to other models, tasks, or hardware.
Best Value
How the approaches compare
| Approach | What changes | Sequence-length scaling | Memory behavior | Evidence scope |
|---|---|---|---|---|
| Standard full self-attention | Nothing; the calculation as originally defined | Runtime and memory quadruple when length doubles (NVIDIA Transformer Engine 2.15.0 documentation) | Forms the full n × n score matrix per head | The baseline that the other two are measured against |
| Memory-optimized exact attention (flash attention, as described by NVIDIA) | Memory handling and data movement, using tiling and recomputation; the output is the same exact attention | Still full pairwise attention, so the quadratic pattern is not removed | Does not store the full softmax matrix for the backward pass; saves normalization factors instead | Described in NVIDIA’s release-specific documentation |
| Linear attention (Katharopoulos et al., 2020) | The attention operation is reformulated with kernel feature maps and associativity | Linear, O(N), as stated in the paper | Avoids forming the n × n score matrix | Paper-specific experiments; the reported speedup applies to autoregressive prediction of very long sequences in that setup |
The three rows are not interchangeable. The first two keep the same mathematical output and differ only in how much memory and data movement they need. The third trades the exact softmax formulation for a different one, so its quality on a given task has to be judged on that task.
The original results and what they measure
The 2017 paper introduced the Transformer as a sequence transduction model built solely on attention, with no recurrence or convolution. Its translation results are historical benchmark numbers for that model on the WMT 2014 test sets:
| Task | BLEU score | Source |
|---|---|---|
| English-to-German, WMT 2014 | 28.4 | Vaswani et al., 2017 |
| English-to-French, WMT 2014 | 41.0 | Vaswani et al., 2017 |
These scores show what the original model achieved on those translation tasks in 2017. They do not predict how a current model would perform on other tasks, and they say nothing about the speed of the attention layer.
Free tools Windows power users keep installed
One-click scans. No signup required.
Why the distinction matters in practice
When someone says a model “handles long context,” the first question is which kind of attention it uses. A model that runs standard full attention still pays the n² cost, although an exact memory-efficient kernel may let it fit longer inputs on the same hardware. A model built on a linear reformulation changes the cost curve, but it has to be evaluated on its own terms. Keeping these categories apart prevents a memory improvement from being read as an asymptotic breakthrough, and an asymptotic claim from being read as a universal speed guarantee.
The core facts are stable: attention is a weighted sum of values, the weights come from learned query–key scores passed through a softmax, and full pairwise attention scales with n². Implementation details, supported features, and speeds for specific software and hardware change between releases, so check the versioned documentation of the library you use before relying on a particular number.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
