Error feedback does not simply discard the part of a gradient that compression removes: it saves that residual and adds it to a later gradient. The classic guarantee is for SGD-like updates, though, and it does not transfer directly to Adam. Adam’s history-dependent, coordinate-wise scaling means that preserving accumulated gradients does not necessarily preserve parameter updates. That is a limitation of the direct extension—not proof that every Adam-and-compression method fails.
What error feedback does in gradient compression
Gradient compression reduces the information communicated during distributed training. A compressor might, for example, keep only coarse signs or quantized values instead of transmitting a full-precision gradient. If the omitted information is discarded at every step, the resulting update can be biased and optimization can suffer.
As an Amazon Associate I earn from qualifying purchases.
Error feedback, also called error compensation, carries the omitted information forward. In a simplified SGD-style description, let g_t be the current gradient, r_{t-1} the saved residual from the previous step, and C the compressor:
Free tools Windows power users keep installed
One-click scans. No signup required.
- Correct: add the saved residual to the current gradient:
u_t = g_t + r_{t-1}. - Compress and apply: send and use
c_t = C(u_t). - Save the new residual: retain the difference,
r_t = u_t - c_t.
The compressed vector can omit information now, but that difference becomes part of the next corrected input rather than vanishing. In this schematic, c_t = g_t + r_{t-1} - r_t, so summing across steps makes the intermediate residuals cancel. With an initial residual of zero, the accumulated compressed vectors differ from the accumulated original gradients by the final residual. This telescoping identity explains the intuition; it is not by itself a convergence proof, and it does not establish equivalence of parameter updates for every optimizer.
#1 Best Overall
Why the classical argument fits SGD better than Adam
In plain SGD, the gradient determines the update through a comparatively direct scaling by the learning rate. Momentum adds a running average, but its update still has a linear structure that makes it possible to relate accumulated gradients and updates. The 2019 paper Error Feedback Fixes SignSGD and other Gradient Compression Schemes analyzes EF-SGD and reports convex counterexamples for signSGD, along with a convergence-rate result for EF-SGD in the paper’s analyzed settings.
The authors state: “We prove that our algorithm (EF-SGD) with arbitrary compression operator achieves the same rate of convergence as SGD without any additional assumptions.” That claim belongs to their EF-SGD analysis and its settings; it is not a guarantee for Adam, every compressor implementation, or every later training setup.
Rank #2
- Use scikit-learn to track an example ML project end to end
- Explore several models, including support vector machines, decision trees, random forests, and ensemble methods
- Exploit unsupervised learning techniques such as dimensionality reduction, clustering, and anomaly detection
- Dive into neural net architectures, including convolutional nets, recurrent nets, generative adversarial networks, autoencoders, diffusion models, and transformers
- Use TensorFlow and Keras to build and train neural nets for computer vision, natural language processing, generative models, and deep reinforcement learning
Adam changes the update rule. It maintains exponential moving averages of gradients and squared gradients, applies bias corrections, then divides the first estimate coordinate by coordinate by the square root of the second estimate plus epsilon. The denominator acts as an adaptive preconditioner: each coordinate’s effective step depends on its gradient history.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsThat history dependence is the key mismatch. A gradient-space residual moved to a later step is not generally equivalent to applying that residual as a parameter displacement, because Adam’s coordinate-wise scaling can change as its moment estimates evolve. Preserving a sum of gradients therefore does not, on its own, preserve Adam’s sequence of parameter updates. The original error-compensation techniques were designed for linearly dependent SGD- or momentum-style updates; the 1-bit Adam paper describes their direct application to nonlinear Adam updates as not working in the same way.
Rank #3
What “Adam breaks it” does—and does not—mean
“Adam breaks error feedback” is shorthand for a specific problem: the standard residual argument and its assumptions do not carry over unchanged to Adam’s adaptive update. It should not be read as a theorem that all compression, all residual mechanisms, or all Adam-family optimizers fail.
- Error feedback can mitigate information loss from biased compressors in settings for which its method and analysis apply.
- Adam’s changing preconditioner means a residual convention that works for SGD needs a separate justification when paired with Adam.
- Some Adam-specific compression algorithms change the method to address that mismatch, and their guarantees apply to those algorithms and assumptions.
How 1-bit Adam handles the mismatch
In their 2021 ICML paper, Hanlin Tang and co-authors propose a specific approach: begin with a warmup phase, then treat Adam’s variance estimate as a fixed preconditioner during the compression phase. They describe their key finding as: “Adam’s variance becomes stable (after a warmup phase) and can be used as a fixed precondition for the rest of the training (compression phase).” Holding that preconditioner fixed changes the setting so compression can be applied without assuming that the usual time-varying Adam scaling behaves like an SGD update.
Rank #4
The authors report the following maximum results for their experiments. These are paper-specific findings, not expected gains for arbitrary models, hardware, or training workloads:
| Reported result | Context in the 2021 paper |
|---|---|
| Up to 5× communication-volume reduction | Maximum reported by Tang and co-authors for 1-bit Adam. |
| Up to 3.3× higher throughput | BERT-Large pretraining experiments. |
| Up to 2.9× higher throughput | SQuAD fine-tuning experiments. |
| Experiments on up to 256 GPUs | Scale of the paper’s reported experiments. |
Communication-volume reduction and end-to-end throughput are different measures: sending fewer bits does not by itself determine how much faster a full training run will be. The throughput figures above are tied to the paper’s named workloads and experiments, while the communication figure describes volume.
Best Value
How the main results differ
These papers address related communication costs, but their results are not interchangeable. Their optimizer structure, compression mechanisms, and theoretical scope differ.
| Work | Optimizer and method | What its result supports |
|---|---|---|
| Karimireddy, Rebjock, Stich, and Jaggi (2019), Error Feedback Fixes SignSGD and other Gradient Compression Schemes | EF-SGD with error feedback for compressed gradients. | Convex counterexamples for signSGD and an EF-SGD convergence-rate result in the settings analyzed by the paper. |
| Tang and co-authors (2021), 1-bit Adam: Communication Efficient Large-Scale Training with Adam’s Convergence Speed | Adam with warmup followed by a fixed variance-based preconditioner during compression. | Paper-specific communication-volume and throughput maxima for BERT-Large pretraining and SQuAD fine-tuning. |
| Chen and co-authors (2021), Quantized Adam with Error Feedback | A distributed method using gradient and weight quantization with error feedback. | Convergence analyses for the paper’s stated stochastic nonconvex setting; the result does not establish a guarantee for other Adam variants or residual conventions. |
| Fatkhullin and co-authors (2025), EF21 with Bells & Whistles: Six Algorithmic Extensions of Modern Error Feedback | EF21 and extensions developed in response to limitations identified in classical error-feedback theory. | A later theoretical treatment that discusses assumptions such as bounded gradients and pessimistic rates in some regimes; it does not show that all classical EF implementations fail in practice. |
What to check before applying error feedback to Adam
A method’s name alone is not enough to establish that its compression and optimizer are compatible. When evaluating a result or implementation, check:
- Update structure: whether the analysis is for SGD, momentum, Adam, or another adaptive optimizer.
- Residual definition: which vector is compressed, what error is saved, and where that residual re-enters the computation.
- Compressor: whether the method uses signs, quantization, or another compression scheme, and at what compression level.
- Guarantee and assumptions: the analyzed objective and stochastic setting, plus conditions such as bounded gradients. Classical EF results are not assumption-free across all settings.
- Metric: whether evidence concerns communication volume, convergence, or end-to-end throughput; these answer different questions.
The available results do not establish convergence guarantees for every Adam-family optimizer paired with every compressor and residual convention. The right conclusion is narrower: direct error-feedback reasoning for SGD does not automatically justify the same construction with Adam, while specific Adam compression methods have their own mechanisms and analyses.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
