Medium

Have a go at it. The editor and the docs panel are open, and your code is saved as you type. Running it needs a free account — you’ll come back to exactly what you wrote.

Inverted Dropout Forward Pass

~18 mincode completion

L1 and L2 penalise weights. Dropout does something stranger: during training it deletes a random subset of activations on every forward pass, so no single unit can be relied upon. The network is forced to spread its representation out.

The catch is what happens at test time. With drop probability $p$ , a unit's expected output during training is $(1 - p)$ times its true activation. If you dropped units during training and did nothing at inference, every downstream layer would suddenly see activations that are $1/ (1 - p)$ times larger than it was trained on.

Inverted dropout fixes this at training time so inference stays a plain forward pass:

eval: \tilde{a} = a

Dividing by $1 - p$ during training restores the expected value, so at inference you do nothing at all. This is what every framework actually implements.

Worked example ( $p = 0.5$ ):

a    = [1.0, 2.0, 3.0, 4.0]
mask = [1,   0,   1,   0  ]      1 = keep, 0 = drop

train:  [1.0, 2.0, 3.0, 4.0] * [1,0,1,0] / (1 - 0.5)
     =  [2.0, 0.0, 6.0, 0.0]
eval:   [1.0, 2.0, 3.0, 4.0]

Your task:

Implement dropout_forward(x, mask, p, training).

x is the activation array (any shape).

mask is a same-shaped array of 1 (keep) and 0 (drop), passed in rather than sampled so the result is reproducible.

p is the drop probability, 0 <= p < 1.

training is a bool. When False, return x unchanged, ignoring the mask entirely.

Return an array of the same shape as x.

Note p = 0: the scale factor is 1/(1-0) = 1, so training and eval agree. That is the sanity check that your scaling is on the right side of the fraction.

Example Tests

Training with p=0.5: kept units are doubled, dropped units zeroed

Input: {"p":0.5,"x":[1,2,3,4],"mask":[1,0,1,0],"training":true}

Expected: [2,0,6,0]

Eval mode returns x unchanged and ignores the mask

Input: {"p":0.5,"x":[1,2,3,4],"mask":[1,0,1,0],"training":false}

Expected: [1,2,3,4]

p=0.2 scales kept units by 1/0.8 = 1.25

Input: {"p":0.2,"x":[10,-4,0,8],"mask":[1,1,0,1],"training":true}

Expected: [12.5,-5,0,10]

Python

import numpy as np

def dropout_forward(x: np.ndarray, mask: np.ndarray, p: float,
                    training: bool) -> np.ndarray:
    """
    Apply inverted dropout.

    Args:
        x:        activation array
        mask:     same-shaped array of 1 (keep) / 0 (drop)
        p:        drop probability in [0, 1)
        training: if False, return x unchanged

    Returns:
        Array of the same shape as x.
    """
    # YOUR CODE HERE
    pass

Loading docs…