DeepSeek V4 pairs two attention paths—Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA)—while using Manifold-Constrained Hyper-Connections (mHC) to shape information flow between layers. DeepSeek Sparse Attention (DSA) is not a third peer path: it selects relevant compressed key-value entries within CSA. In short, CSA and HCA handle attention in complementary ways; mHC handles residual-stream mixing.
How the four mechanisms fit together
The simplest way to understand the stack is to separate attention over past tokens from information flow between model layers. DeepSeek’s V4 model card describes CSA and HCA as a hybrid attention design. CSA compresses the key-value (KV) cache along the sequence dimension and applies DSA to select entries. HCA compresses the sequence more heavily and uses dense attention over the resulting representation. Separately, mHC constrains how residual streams are mixed across layers.
As an Amazon Associate I earn from qualifying purchases.
- CSA: a less heavily compressed sequence representation with sparse, indexed selection.
- DSA: the selection mechanism used within CSA to choose entries for core attention.
- HCA: a more heavily compressed representation attended to densely.
- mHC: a residual-connection design that constrains inter-layer mixing; it does not select historical tokens.
DeepSeek describes these mechanisms in its V4 model card; implementation details for CSA and HCA are also documented by Hugging Face Transformers.
What DSA does—and what its complexity claim means
DeepSeek Sparse Attention uses a learned “lightning indexer” to score preceding KV entries for a query. A top-k selector then retains a subset for the core attention operation. Rather than computing core attention over every preceding entry, the operation attends to the selected entries.
#1 Best Overall
DeepSeek’s V3.2 report describes the core attention complexity as changing from O(L²) to O(Lk), where L is sequence length and k is the number of selected entries. That qualification matters: the report says the indexer itself still has O(L²) complexity. The claim therefore applies to the core attention operation, not to all computation involved in DSA. See the DeepSeek V3.2 technical report.
CSA and HCA: two different compression-and-attention tradeoffs
Both paths reduce the sequence representation used by attention, but they differ in how much they compress it and whether they select entries sparsely or use them all densely. The comparison below reflects the architectural descriptions in DeepSeek’s model card and Transformers documentation, not a measured performance ranking.
| Mechanism | KV sequence treatment | Selection behavior | Architectural emphasis |
|---|---|---|---|
| CSA | Compresses along the sequence dimension; the Transformers description specifies lower compression and overlapping windows. | A Lightning Indexer gathers top-k entries before core attention. | Selective access to a less-compressed pool. |
| HCA | Uses heavier compression. | No indexer is used in the documented path; each pooled entry participates in dense attention. | Broad attention across a more-compressed pool. |
These are complementary attention routes in V4, not competing labels for the same single mechanism. The sources describe their design, but do not establish a universal speed, quality, or memory advantage for one route over the other.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesWhat mHC changes
Manifold-Constrained Hyper-Connections address residual-stream structure rather than attention over the token history. DeepSeek describes mHC as constraining residual mapping to the manifold of doubly stochastic matrices, also called the Birkhoff polytope. The stated aim is to stabilize signal propagation while retaining expressivity. Transformers documentation describes parallel residual streams mixed through a doubly stochastic projection.
Put another way, CSA and HCA determine how attention accesses compressed sequence information, while mHC constrains how representations are mixed as they pass between layers. It should not be described as another sparse-attention method.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How to interpret V4’s context specification and performance claims
DeepSeek’s V4 model card, published April 27, 2026, specifies a 1M context length. That is a model-card specification, not an independent benchmark result or a guarantee that every deployment configuration supports the full context. Actual availability can depend on the serving setup.
Likewise, architectural descriptions and complexity statements are not end-to-end performance measurements. The cited DSA complexity reduction is specifically for core attention and leaves the indexer’s O(L²) complexity intact. A later StreamIndex study examines memory-bounded CSA indexer processing using synthetic V4-shaped indexer-step experiments; it disclaims claims about end-to-end performance on real checkpoints. Its results should not be generalized into a production performance claim.
Recommended Free Tools
For the architecture, the useful mental model is: DSA selects within CSA; CSA and HCA form V4’s hybrid attention design; and mHC constrains residual information flow across layers.
Quick Recap
Best Value
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
