There is no universally best learning rate for either SGD or Adam. Use published defaults only as starting points, compare candidate rates on your own validation task while holding other settings constant, and choose based on stability, useful training progress, and validation performance. Adam adapts updates per parameter, but it still requires a global learning-rate choice.
What the learning rate controls
The learning rate sets the scale of an optimizer’s parameter updates. If it is too large for a particular model and task, training may behave erratically or fail to make useful progress; if it is too small, progress may be limited within the available training budget. The practical question is not which rate is best in the abstract, but which tested configuration works well for your task.
As an Amazon Associate I earn from qualifying purchases.
Adam uses estimates of first and second gradient moments to adapt its updates for individual parameters. That adaptivity does not eliminate the global learning-rate parameter exposed by Adam implementations. SGD uses stochastic gradients with its configured rate and may also use momentum. Because their update mechanisms differ, the same numeric learning rate should not be assumed to work equally well for both.
Choose a baseline, not a presumed optimum
Adam
PyTorch’s current documentation lists 1e-3 as Adam’s default learning rate and (0.9, 0.999) as its default beta values; these are implementation defaults and can vary by framework version or implementation. See the PyTorch Adam documentation.
#1 Best Overall
The original Adam paper reports α=0.001, β1=0.9, β2=0.999, and ε=10-8 as good default settings for the machine-learning problems tested. Those settings are useful context, not a guarantee of the best rate for another model or dataset. See Kingma and Ba’s 2014 Adam paper.
SGD
Set an explicit starting learning rate for SGD, then test alternatives on the target task. The evidence here does not establish one general-purpose numerical SGD rate, so do not treat a value from a different model or workflow as a universal standard.
Rank #2
- Use scikit-learn to track an example ML project end to end
- Explore several models, including support vector machines, decision trees, random forests, and ensemble methods
- Exploit unsupervised learning techniques such as dimensionality reduction, clustering, and anomaly detection
- Dive into neural net architectures, including convolutional nets, recurrent nets, generative adversarial networks, autoencoders, diffusion models, and transformers
- Use TensorFlow and Keras to build and train neural nets for computer vision, natural language processing, generative models, and deep reinforcement learning
A controlled procedure for comparing rates
- Fix the evaluation setup. Set aside validation data and choose a metric that reflects the task. Keep the model architecture, initialization, data processing, batch size, and training budget fixed across candidate runs. Set seeds where the framework and workload permit; if randomness makes close results hard to distinguish, repeat the comparison.
- Choose starting values deliberately. For Adam, use the documented
1e-3default as a baseline if appropriate for your framework and version. For SGD, choose an explicit experimental starting point rather than assuming Adam’s value transfers. - Test clearly different candidate rates. Run a modest set of rates that differ in scale, changing no other experimental variable. There is no universally established candidate grid or multiplier. Record each run’s validation metric, training stability, and progress under the same budget.
- Reject unstable runs and compare the rest. Do not favor a rate merely because its training metric improves quickly. Reject rates that make training unstable or cause divergence; among viable runs, compare useful progress and validation results.
- Inspect training and validation curves together. Training progress alone does not show whether a model performs well on held-out data. Use the validation metric as part of the selection decision, and note whether it stops improving while training continues.
- Retest a final schedule or rate. If you change the schedule or training trajectory, compare the resulting configuration with the baseline using the same evaluation protocol. Report what performed best in the tested setup, rather than claiming it is optimal for other tasks.
When to change the learning rate during training
A fixed learning rate is not the only option. TensorFlow’s guide describes schedules keyed to epoch or batch count, as well as dynamic changes based on validation behavior. It gives exponential, piecewise constant, polynomial, and inverse-time schedules as examples, and identifies ReduceLROnPlateau as a callback that changes the current rate when validation loss stops improving. The guide also notes a common pattern of gradually reducing the learning rate as training progresses. See TensorFlow’s training and evaluation guide.
Keras accepts schedule objects as an optimizer’s learning-rate argument; see the Keras learning-rate schedules documentation. Choose a schedule for the behavior you want to test. The available evidence does not establish one universally best schedule.
Rank #3
How to compare SGD with Adam fairly
If deciding between the optimizers as well as tuning their rates, evaluate both against the same task-specific criteria. Keep the data split, model, training budget, and evaluation protocol comparable, and tune the learning rate for each optimizer rather than assuming one shared value is fair.
- Validation performance: Compare the metric that matters for the task.
- Stability: Note erratic behavior or divergence, not just the final score.
- Useful progress: Compare how much progress each configuration makes under the same compute or step budget.
- Sensitivity: Record whether results change substantially with the starting rate or schedule.
- Reproducibility: Keep the training and evaluation conditions aligned so the comparison is meaningful.
These criteria help make a decision for a specific setup; they do not imply a universal winner between SGD and Adam.
Rank #4
What to report so the choice can be reproduced
State the framework and version, optimizer, initial learning rate, schedule and its parameters, batch size, training budget, and validation criterion. Include the tested configuration and the validation result. Framework API defaults are version-sensitive, so naming only “Adam default” is not enough to make a result reproducible.
Quick Recap
Best Value
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
