Hard

Adam Optimizer Update

Hard

~25 min

code completion

Adam: Adaptive Moment Estimation

Adam combines momentum (first moment) and RMSprop (second moment) with bias correction:

m \leftarrow β_{1} m + (1 - β_{1}) g

v \leftarrow β_{2} v + (1 - β_{2}) g^{2}

\overset{m}{^} = \frac{m}{1 - β _{1}^{t}}, \overset{v}{^} = \frac{v}{1 - β _{2}^{t}}

w \leftarrow w - \frac{α m ^}{v ^ + ϵ}

The bias correction ( $\overset{m}{^}$ , $\overset{v}{^}$ ) accounts for the fact that $m$ and $v$ are initialized at zero and are biased toward zero in early steps.

Typical defaults: $β_{1} = 0.9$ , $β_{2} = 0.999$ , $ϵ = 1 0^{- 8}$ , $α = 0.001$ .

Your task:

Implement adam_update(weights, m, v, gradient, learning_rate, beta1, beta2, epsilon, t). Return (weights_new, m_new, v_new).

Example Tests

First step, cold start: weights decrease (accessor [0])

Input: {"m":[0],"t":1,"v":[0],"beta1":0.9,"beta2":0.99,"epsilon":1e-8,"weights":[0],"gradient":[1],"learning_rate":0.001}

Expected: [-0.001]

Zero gradient: weights unchanged (accessor [0])

Input: {"m":[0,0],"t":1,"v":[0,0],"beta1":0.9,"beta2":0.99,"epsilon":1e-8,"weights":[1,-1],"gradient":[0,0],"learning_rate":0.01}

Expected: [1,-1]

m_new correct (accessor [1])

Input: {"m":[0],"t":1,"v":[0],"beta1":0.9,"beta2":0.99,"epsilon":1e-8,"weights":[0],"gradient":[1],"learning_rate":0.001}

Expected: [0.1]

You can read the full problem statement above. Create a free account to run code in the browser, submit solutions, and track your progress.