Free tools Windows power users keep installed
One-click scans. No signup required.
Researchers are testing ways to process sequences without relying on a standard Transformer attention stack. Mamba, RWKV and Hyena take different routes toward more efficient sequence modeling, while translation and reinforcement-learning studies show how these ideas can also be combined with attention or applied beyond text. The results are promising, but they do not establish that Transformers have been displaced.
What does “post-transformer” mean?
Here, “post-transformer” describes an active search for alternatives to the standard Transformer architecture—not a world without language models, or proof that Transformers are obsolete. The approaches differ in how they carry information through a sequence: some update a compact state, some use recurrent-style computation, and others replace attention with long convolutions and gating. Hybrid models keep attention in the mix.
As an Amazon Associate I earn from qualifying purchases.
Those choices matter because attention can be expensive for long sequences, while a cheaper sequence mechanism may make different trade-offs in recall, quality, training, or inference. A claim about speed or quality belongs to the model, task, hardware and evaluation in the paper that measured it; it is not a universal guarantee.
How do Mamba, RWKV and Hyena differ?
| Approach | How it processes sequences | Evidence in the cited paper |
|---|---|---|
| Mamba | Selective state-space updates that depend on the input | Language, audio and genomics experiments |
| RWKV | Parallelizable training with recurrent-style inference | Language models evaluated up to 14 billion parameters |
| Hyena | Long convolutions interleaved with data-controlled gating | Language modeling on WikiText103 and The Pile, plus operator-speed comparisons at specified sequence lengths |
Mamba: selectively carry or forget information
In a conventional state-space model, the dynamics do not depend on the current input. Albert Gu and Tri Dao’s 2023 Mamba paper treats that as a limitation for discrete, content-dependent inputs such as language. Mamba makes model parameters functions of the input, allowing it to selectively propagate or forget information, and introduces a hardware-aware recurrent algorithm. The authors describe its architecture as lacking both attention and MLP blocks.
#1 Best Overall
The authors report linear scaling with sequence length and 5× higher inference throughput in their paper’s abstract. They also report that Mamba-3B outperformed same-size Transformers and matched Transformers twice its size on the paper’s pretraining and downstream evaluations. These are results from those experiments, not a promise for arbitrary hardware, software stacks or tasks. Read the Mamba paper.
RWKV: train in parallel, infer recurrently
RWKV combines parallelizable training with inference formulated as an RNN. Its authors describe a linear-attention mechanism that can be formulated as either a Transformer or an RNN; in their formulation, inference has constant computational and memory complexity. The 2023 paper reports training models up to 14 billion parameters and performance on par with similarly sized Transformers in its evaluations. That comparison applies to the models and tests in the paper, not every RWKV variant or present-day task. Read the RWKV paper.
Hyena: long convolutions with gating
Hyena interleaves implicitly parameterized long convolutions with data-controlled gating as a subquadratic alternative to attention. On WikiText103 and The Pile, the authors report Transformer-quality language modeling with a 20% reduction in training compute at sequence length 2k. For Hyena operators versus highly optimized attention, they report being 2× faster at sequence length 8k and a 100× speedup at 64k. Those figures describe the paper’s specified comparisons and lengths; they should not be read as universal speedups for all models or systems. Read the Hyena paper.
Does attention still help?
Yes, at least in some tested settings. A WMT 2024 machine-translation comparison evaluated RetNet, Mamba and hybrid Mamba models on sentence- and paragraph-level datasets. Mamba was highly competitive with Transformers in those experiments, while adding attention improved translation quality, robustness to sequence-length extrapolation and named-entity recall.
That result complicates any claim that attention is simply obsolete. A model may benefit from an alternative sequence mechanism and still use attention where it helps; the study supports that conclusion for its tested translation setups, not every language or application. Read the WMT 2024 study.
Are these ideas being used beyond language generation?
A 2024 ICML study applied RetNet to a token-based reinforcement-learning world model called REM. It added Parallel Observation Prediction and evaluated the agent on Atari 100K. The authors report 15.4× faster imagination than prior token-based world models in their study, and superhuman performance on 12 of the 26 Atari 100K games they tested.
This is evidence that recurrent-style sequence architectures are being explored for world models, not evidence of widespread deployment or a general advantage across reinforcement learning. Read the REM paper.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
How should you judge a possible Transformer substitute?
There is no single winner established by these papers. A useful comparison starts with the job the model must do, then checks whether a claimed improvement survives a like-for-like test.
Best Value
- Match the task and modality. Language modeling, translation, audio, genomics and reinforcement-learning world models are different tests. A result in one does not establish performance in another.
- Compare models at a matched scale. Parameter count and evaluation setup affect quality comparisons. Note whether a result is against same-size models or larger ones.
- Separate training from inference. Parallel training, decode throughput, memory use and recurrent state requirements are distinct concerns. A gain in one does not automatically imply a gain in the others.
- Check the context lengths actually tested. Theoretical scaling and measured quality at long lengths are not the same. Look for sequence-length extrapolation results and exact-recall tests, such as named-entity recall.
- Treat paper benchmarks as evidence, not adoption data. The cited papers report research results; they do not provide an industry-wide measure of how widely these architectures are deployed.
The emerging picture is not a clean handoff from Transformers to one replacement. It is a set of architecture-specific experiments—some with impressive efficiency results, some competitive on particular tasks, and some showing that attention can still improve a hybrid. Which design is useful depends on the workload and the evidence for that workload.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
