Vanilla SGD is just the beginning. Adam, RMSprop, and momentum methods adapt learning rates and escape sharp minima faster.
SGD with Momentum
~20 min· Hard
RMSprop Update
Adam Optimizer Update
~25 min· Hard
Bias-Corrected Exponential Moving Average (the Heart of Adam)
~15 min· Easy
Sign in for the concept check
Optional multiple-choice questions on the ideas behind this section. Most useful after you have tried the coding problems above.