Vanilla SGD is just the beginning. Adam, RMSprop, and momentum methods adapt learning rates and escape sharp minima faster.
SGD with Momentum
~20 min· Hard
RMSprop Update
Adam Optimizer Update
~25 min· Hard
Bias-Corrected Exponential Moving Average (the Heart of Adam)
~15 min· Easy
One AdamW Step
~15 min· Medium
Learning Rate Scaling and Training Cost
~8 min· Intro
Learning Rate from a Range Test
Gradient Clipping by Global Norm
~10 min· Easy
One SGD Step for Matrix Factorization
Sign in for the concept check
Optional multiple-choice questions on the ideas behind this section. Most useful after you have tried the coding problems above.