Adam Optimizer Explained: Why It Is Deep Learning's Default

"...straightforward to implement, is computationally efficient, has little memory requirements..." - Kingma and Ba (Adam, ICLR 2015)

Almost every neural network you will train starts with the same line of code: optimizer = Adam(...). Very few people who type it could write the update rule.

This week we're talking about the Adam optimizer, the default way deep learning models learn, and yes, this can get you a very comfortable job.

Here's the idea: the Adam optimizer is a version of gradient descent that gives every parameter its own step size. It keeps two running averages for each weight, one of the gradient and one of the squared gradient, and scales each step by their ratio. Parameters with large, noisy gradients take small steps, and parameters with small, consistent gradients take larger ones.

The quote above is from the 2014 abstract, and it aged in an interesting way. For a single weight, Adam's two extra numbers are nothing. For a large language model trained in mixed precision, the ZeRO paper counts 16 bytes of training memory per parameter, and 12 of those are the optimizer's copy of the weights and its two running averages. That is why the biggest training runs obsess over optimizer memory, and why the first serious challenger at scale is recent: Moonshot AI trained its trillion-parameter Kimi K2 on 15.5 trillion tokens with MuonClip, a Muon-based optimizer, instead of AdamW. Adam is still the default. It is no longer unquestioned.

How does the Adam optimizer work?

For each parameter, Adam keeps a moving average m of the gradient (the momentum) and a moving average v of the squared gradient (how large the gradient usually is). Both start at zero, so both are corrected for that start, and then the step is $${\theta }_t={\theta }_{t-1}-\alpha \frac{{\hat{m}}_t}{\sqrt{{\hat{v}}_t}+ϵ}$$ where the hats mean bias-corrected. The defaults from the paper, a learning rate of 0.001 with decays of 0.9 and 0.999, still work surprisingly often. The division is the whole trick: it makes the size of each step roughly independent of the scale of that parameter's gradient.

Does the Adam optimizer change the learning rate?

Yes, per parameter, but not the one you set. Adam divides by the recent gradient magnitude, so every weight gets its own effective step, and the learning rate you choose acts as a rough upper bound on how far any weight moves per step. It does not replace a schedule: large models still use warmup and decay on top of Adam.

Gem: the bias correction matters most at the start. With a decay of 0.999, the uncorrected squared-gradient average begins about 1,000 times too small, so the first steps would be enormous. The free challenge this week is exactly that correction.

Adam vs SGD vs AdamW

The weight decay row is the one interviewers love. Loshchilov and Hutter showed in 2019 that adding L2 regularization to Adam's gradient gets divided by the same adaptive denominator, so weights with large gradients are barely regularized. AdamW applies the decay to the weights directly, which is why it is the default for training transformers today.

Now I'm going to explain what this means for you:

The Skill Employers EXPECT

Learn what your optimizer is doing. Deep learning interviews assume you can:

• Write the Adam update, including bias correction, from memory
• Use the Adam optimizer in PyTorch, set its learning rate, betas and weight decay, and attach a schedule
• Explain the difference between Adam and AdamW in one sentence
• Read a loss curve and tell a learning rate that is too high from one that is too low
• Explain why Adam's memory cost matters when a model has billions of parameters

If you have only ever called Adam(model.parameters()), you have never tuned it, you have hoped.

The Skill That Separates You

Learn to diagnose training through the optimizer.

When a run diverges at step 40,000, strong engineers look at optimizer state before they touch the model. You should know how to:

• Spot ill-conditioning, where features on wildly different scales stall plain gradient descent, and show Adam fixing it
• Pick epsilon and the second decay deliberately (LLM runs often lower 0.999 to 0.95 for stability)
• Add warmup so the first steps do not blow up, and gradient clipping for the spikes that come later
• Estimate optimizer memory for a model and know what sharding it (ZeRO) and 8-bit optimizer states buy you
• Know when to benchmark a newer optimizer such as Muon instead of assuming Adam

The Project To Learn These Skills This Week

Replicate the Adam paper in NumPy and reproduce its headline result: adaptive steps beat plain SGD on a badly conditioned problem. This is a graded Pro project on GRADuateML: Adam, from the paper. You get a logistic regression whose features range in scale from 0.01 to 10, the same 300-step budget for every optimizer, and a hidden test set of 1,200 rows. No autograd and no library optimizers. Not on Pro? The free challenge below is Adam's bias correction, by hand.

Requirements (each one is a graded milestone):

• Measure how badly conditioned the feature matrix is.
• Derive the logistic loss and its gradient in NumPy.
• Train plain minibatch SGD as the baseline: ROC-AUC of at least 0.80.
• Add per-parameter scaling with RMSProp: ROC-AUC of at least 0.86.
• Implement Algorithm 1 from the paper exactly, bias correction included, and match the reference update.
• Train with Adam and beat both baselines: ROC-AUC of at least 0.87 and log loss of 0.44 or lower.
• Write up the replication and explain why the ordering came out the way the paper says.

This project teaches a practical lesson for production ML:

The optimizer is part of the model. Treat its defaults as hypotheses, not facts.

If you want to sharpen your machine learning skills even more, I also selected a free challenge problem for you this week. It is the bias-corrected moving average at the heart of Adam:

If you learned something from this newsletter, make sure to forward it to a friend.

GRADuateML

The full Adam paper replication, graded against hidden data, is part of GRADuateML Pro.