What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Yes—Dust demonstrates that a transformer can be pretrained without backpropagation, but the October 2026 results do not show that it is a practical replacement. Dust estimates updates from forward-pass loss feedback, and its test-loss comparison with backprop changes with the token budget and model size. Its authors say it is not yet compute-efficient enough to replace backprop.
How Dust estimates updates without backpropagation
Backpropagation computes how a change in each model weight would affect the loss. Dust instead uses a zeroth-order approach: it perturbs layer activations, runs a forward pass, and uses the resulting token-level losses as feedback about those perturbations.
As an Amazon Associate I earn from qualifying purchases.
For each linear layer, Dust adds Gaussian noise to the layer output independently at each token. Perturbations associated with lower loss receive a more favorable reward; those associated with higher loss are penalized. Averaging the reward-weighted noise estimates an error at the layer output. Combining that estimate with the layer input through an outer product gives an estimate of the weight update.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallAttention uses a related procedure to estimate attention-output error across current and future tokens. Most weights use these activation-space perturbations; the paper says residual mixing scalars are trained with ordinary weight-space evolution strategies.
#1 Best Overall
The “virtual population”
Evolutionary methods often evaluate many perturbed candidate models. Dust instead treats token positions as parallel members of a “virtual population”: each token gets an independent perturbation, and the forward pass supplies loss feedback without materializing a separate model for every candidate. That design is central to the method, but it does not by itself establish that Dust is more compute-efficient than backprop.
What was tested
Q Labs’ main experiment trained GPT-style transformers on FineWeb with a 4096-token BPE tokenizer. Each batch contained 16,000 tokens: eight sequences of 2048 tokens. The runs used one epoch of data and SGD with momentum at a constant learning rate. The base model was eight layers wide 512. The authors tuned settings separately by token budget and population, and reported three seeds per experimental cell.
Rank #2
Evaluation used held-out validation and test sets of 544 sequences each. Reported test loss is from the checkpoint selected using validation loss. The appendix describes the base decoder-only model as 37.7 million parameters, with eight pre-norm blocks, width 512, eight attention heads, RMS normalization, rotary position embeddings and a squared-ReLU MLP of width 2048. A separate model-size study covered configurations from 2.0 million to 243 million parameters.
Free tools Windows power users keep installed
One-click scans. No signup required.
How Dust compared with backprop
The base-model results do not show a consistent winner. At a 1 million-token budget, Dust with 16,000 draws had slightly lower test loss; at 10 million tokens, backprop had lower test loss. These are Q Labs’ averages across three seeds at validation-selected checkpoints.
Rank #3
| Training-token budget | Dust, 16k draws | Backprop | Lower reported test loss |
|---|---|---|---|
| 1 million | 5.916 | 5.959 | Dust |
| 10 million | 5.049 | 4.989 | Backprop |
In the model-size comparison at 10 million tokens and 16,000 draws, Dust was marginally lower at the two smallest sizes, while backprop was lower at the larger sizes. The figures are Q Labs’ reported test losses; lower is better.
| Model size | Dust | Backprop | Lower reported test loss |
|---|---|---|---|
| 2.0M parameters | 5.171 | 5.180 | Dust |
| 7.3M parameters | 5.065 | 5.066 | Dust |
| 38M parameters | 5.036 | 5.015 | Backprop |
| 243M parameters | 5.086 | 5.048 | Backprop |
These comparisons show feasibility under the reported setup, not a general advantage. They cover short token budgets, and the observed differences are small; the results do not establish how the methods compare at larger training budgets or across other architectures.
Rank #4
What the EGGROLL efficiency claim means
Q Labs estimates that from 1 million tokens upward, Dust can be roughly 1,000 to 10,000 times more efficient than the authors’ transformer implementation of EGGROLL. This is not a general-purpose efficiency ratio or a measured comparison against backprop. In the appendix, population comparisons are obtained by fitting and extrapolating measured curves, so the estimate depends partly on that extrapolation.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →What the 1-billion-token result does—and does not—show
Q Labs reports that Dust’s gradient estimates align more closely with backprop as population grows, with gradient-alignment measurements reaching a 1-billion-token checkpoint. That is a diagnostic of estimated gradients, not evidence that Dust itself completed a 1-billion-token pretraining run. The main pretraining comparison goes up to 20 million tokens.
Is Dust ready to replace backprop?
No. The authors, Samip Dahal, Bishwas Mandal, Serdar Gülbahar and Akshay Vegesna, state: “We do not attempt to make it compute-efficient enough to replace backprop today.” The paper presents Dust as an experimental method and a possible route toward search-based credit assignment, not as a training recipe practitioners should switch to now.
The authors also say that training new kinds of neural networks that backpropagation handles poorly, and improving compute efficiency enough to replace backprop, are both left to future work. Results on larger training budgets, larger models and other architectures remain unresolved; the paper and appendix document author-run experiments, not independent replication.
Quick Recap
Sources and implementation
- Q Labs, “Dust: Pretraining Transformers Without Backpropagation” (October 2026) — method, main experiment, results and author-stated limitations.
- Q Labs’ official appendix (October 2026) — architecture, detailed test-loss tables, tuning and population extrapolation, and gradient-alignment material.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
