Recommended Free Tools
In this controlled test, which mechanisms a hybrid stack contains mattered more than the order they were placed in, as long as each mechanism stayed spread across depth. A balanced alternative ordering landed within 0.16% of the reference on validation loss, while grouping mechanisms by depth or repeating one mechanism throughout cost more. The largest single removal penalty, 2.14%, came from taking out the state-space (linear-recurrent) mixer. Two qualifications apply throughout: the removal test ran on a four-mechanism, 700.9M-parameter proxy rather than the seven-slot flagship, and the placement results were not repeated at the larger 1.514B scale.
What the paper asked
A hybrid language model that mixes several kinds of sequence mixer can differ in two ways: which mechanisms it includes (composition) and where each one sits from the bottom to the top of the stack (placement). The paper “Placement Is Free, Composition Is Not,” by VIDRAFT AI Research (arXiv:2609.20269, revised as v3 on 2026-10-01), tries to separate the two. Its abstract describes the outcome as “a clear three-way dissociation” among composition, placement schedule, and depth distribution.
As an Amazon Associate I earn from qualifying purchases.
How the Latin square lays out seven mixers
A Latin square is a grid in which every symbol appears exactly once in each row and exactly once in each column. The flagship, Aether-7B-5Attn, uses a 7×7 square. Each of its seven mixers occurs once in every row and once in every column, so across the 49 layers no mixer is tied to a single position. The construction balances each mixer’s position across depth, but it does not balance which mixer tends to follow which, so adjacent-pair effects are left uncontrolled.
Free tools Windows power users keep installed
One-click scans. No signup required.
The seven slots are defined at two levels: five base structures and two variants built from them.
#1 Best Overall
- Use scikit-learn to track an example ML project end to end
- Explore several models, including support vector machines, decision trees, random forests, and ensemble methods
- Exploit unsupervised learning techniques such as dimensionality reduction, clustering, and anomaly detection
- Dive into neural net architectures, including convolutional nets, recurrent nets, generative adversarial networks, autoencoders, diffusion models, and transformers
- Use TensorFlow and Keras to build and train neural nets for computer vision, natural language processing, generative models, and deep reinforcement learning
| Slot | Role in the stack |
|---|---|
| Full attention | Base structure |
| Sliding attention | Base structure |
| Differential attention | Base structure |
| Linear-recurrent (Mamba-style) | Base structure; belongs to the state-space-model family |
| NSA | Base structure |
| Compress | Variant; an NSA branch |
| Hybrid | Variant; NSA combined with differential attention |
The flagship is therefore not a set of seven attention-only variants. It includes a recurrent, state-space-model design alongside the attention structures.
The flagship and the proxy
The flagship has 6.59B total parameters, a mixture-of-experts design with roughly 2.98B active parameters, and 49 layers. Its name says 7B, but the paper’s total count is 6.59B. Repeated ablation of the flagship was too costly, so the controlled experiments used a parameter-matched proxy and a single larger check.
Rank #2
| Property | Flagship | 700.9M proxy | 1.514B check |
|---|---|---|---|
| Total parameters | 6.59B (about 2.98B active) | 700.9M | 1.514B (2.16 times the proxy) |
| Layers | 49 | 16 | Not stated in the paper |
| Mechanisms | 7 | 4 | Not stated in the paper |
| Seeds per arm | Not applicable; not ablated | 8 | Not stated in the paper |
| Placement arms | Latin square (the design under test) | Latin-square reference, balanced periodic, contiguous depth bands, homogeneous stack | Not repeated |
| Composition tests | Not ablated | Single-mechanism removals (full, sliding, differential, SSM family) | Homogeneous stack and SSM-family removal |
Placement: where each mixer sits
In the proxy, four arrangements of the same four mechanisms were compared. Each figure is the change in validation loss relative to the Latin-square reference.
| Arrangement | Validation-loss difference vs. reference | What it shows |
|---|---|---|
| Latin-square reference | Baseline | Each mechanism is distributed across depth |
| Balanced periodic schedule | 0.16% change | A different balanced ordering landed almost on top of the reference |
| Contiguous depth bands | 0.59% penalty | Grouping mechanisms into depth bands costs loss |
| Homogeneous stack (one mechanism repeated through depth) | 1.68% penalty | The largest placement-type penalty in the proxy |
The authors read these numbers as showing that, when mixers stay distributed, the specific permutation matters little, while clustering and homogeneity matter more. Their abstract puts it this way: “These findings show that performance depends primarily on balanced composition distributed across depth, not on any specific permutation.” That is a finding about this proxy and these four arms, not a guarantee that every ordering performs the same.
Removing one mechanism at a time
The composition test removed one mechanism at a time from the four-mechanism proxy. The seven-slot flagship was not run through this test.
| Mechanism removed | Reported effect in the 700.9M proxy |
|---|---|
| Full attention | Small change |
| Sliding attention | Small change |
| Differential attention | Small change |
| SSM family (Mamba-2) | 2.14% penalty |
The authors interpret the SSM-family result as evidence that this distinct mechanism family mattered in this setup. Each attention-variant removal left three other mechanisms in the stack, so the small changes describe what removing one option did within this set. They do not rank attention variants in general, and they do not show that a Mamba-style mixer is always required.
Rank #4
Does the composition result hold at 1.514B?
The larger check covered only composition, and it repeated two of the proxy measurements.
| Measure | 700.9M proxy | 1.514B model |
|---|---|---|
| Homogeneous-stack penalty | 1.68% | 2.63% |
| SSM-family removal penalty | 2.14% | 3.20% |
| Balanced periodic and depth-band placement | 0.16% change; 0.59% penalty | Not repeated |
Both penalties were larger at 1.514B in the paper’s comparison. Placement was not rerun at that size, and neither placement nor composition has been ablated at the flagship’s seven-mechanism, 6.59B scale. The authors explicitly caution against assuming these results carry over to the seven-mechanism model.
Best Value
Limits to keep in view
- Most of the evidence comes from one 700.9M proxy with four mechanisms and 16 layers. The flagship’s seven-mechanism configuration was not ablated.
- Placement results do not extend to 1.514B.
- The Latin square balances position, not transitions. In the 4×4 proxy, only 7 of the 12 possible ordered adjacent pairs of mixers occur, and some pairs recur three times while others appear once.
- Placement and removal effects are reported as validation-loss differences, so they say nothing direct about downstream task accuracy.
Speed and memory at long context
The paper also benchmarks prefill latency for an isolated mixer layer. These timings cover one layer, not a whole model, so they are not end-to-end throughput.
| Prefill length | Full attention | Sliding attention |
|---|---|---|
| 2K tokens | 0.4 ms | 0.6 ms |
| 8K tokens | 1.5 ms | 1.9 ms |
| 32K tokens | 13.6 ms | 7.6 ms |
Sliding attention is slower at 2K and 8K in these measurements and faster at 32K, which is the basis for the paper’s reading that it scales better at long context. The paper also reports peak memory at the same lengths. Compare these figures only with each other, since they come from the paper’s own setup.
Causal-safety audit
The paper reports a causal-safety check across all 49 layers of the flagship. Injected faults serve as a positive control, showing that the audit can flag a violation when one is present.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteTraining run and release
The paper reports training the flagship on 16 NVIDIA B200 GPUs in a two-node FSDP setup, for 162,000 steps and 144.2B token-samples. The reported training window runs from 2026-05-30 to 2026-07-16, about 46 days, with a final stage of roughly 11,700 B200-hours. These are the authors’ figures for their own run, not a sizing guide for other projects.
According to the paper, the release includes:
- Model weights and architecture source code, licensed under Apache-2.0
- The training-data recipe and tokenizer script
- Training code, launch scripts, and hyperparameters
- The complete training log and evaluation code
- Intermediate checkpoints
Corpus components keep the licenses of their source repositories, so Apache-2.0 does not automatically apply to the underlying data.
Quick Recap
What it means if you build hybrid stacks
- Keep each mixer distributed through depth rather than grouping it into bands. The proxy penalized banded arrangements.
- Keep a linear-recurrent (state-space) mixer in the candidate set. In the four-mechanism proxy, removing it was the costliest single removal tested.
- Verify placement choices at your own model size before adopting them, since the paper does not test placement beyond the proxy.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
