Have a go at it. The editor and the docs panel are open, and your code is saved as you type. Running it needs a free account — you’ll come back to exactly what you wrote.
Derivative of the Sigmoid
The sigmoid squashes any real number into (0,1):
Its derivative is unusually convenient, and worth deriving once so it stops looking like magic:
The derivative is expressible in terms of the output. During backpropagation you have already computed in the forward pass, so the backward pass costs one multiply and no exponentials. Every framework exploits this.
It also explains the vanishing gradient. The maximum of is at , giving 0.25 — so even in the best case a sigmoid layer shrinks the gradient to a quarter. Stack ten of them and you have . Out at z=6, and the derivative is about 0.0025: the unit is saturated and learning has stopped.
z = 0 sigma = 0.5 sigma' = 0.25 <- the maximum z = 2 sigma = 0.8808 sigma' = 0.1050 z = 6 sigma = 0.9975 sigma' = 0.0025 <- saturated
Your task:
Implement sigmoid_derivative(z) for an array z, returning elementwise as an array.
Example Tests
The derivative peaks at zero, where it equals a quarter
Input: {"z":[0]}
Expected: [0.25]
It is symmetric: +2 and -2 shrink the gradient identically
Input: {"z":[-2,0,2]}
Expected: [0.10499,0.25,0.10499]
A saturated unit has almost no gradient left
Input: {"z":[6,-6]}
Expected: [0.00247,0.00247]
import numpy as np
def sigmoid_derivative(z):
"""
Elementwise derivative of the logistic sigmoid.
Args:
z: array of pre-activations, any shape
Returns:
array of the same shape: sigma(z) * (1 - sigma(z))
"""
# YOUR CODE HERE
pass