Medium

Have a go at it. The editor and the docs panel are open, and your code is saved as you type. Running it needs a free account — you’ll come back to exactly what you wrote.

Gradient of the Least-Squares Loss

~15 mincode completion

Everything in this track now assembles into the one gradient you will write most often. For

L (w) = \frac{1}{n} ∥ X w - y ∥^{2}

the gradient with respect to $w$ is

Read it as a chain rule on vectors. The residual $r = X w - y$ is the inner function; $∥ r ∥^{2}$ differentiates to $2 r$ ; and is the Jacobian of $r$ with respect to $w$ , applied on the left. Three pieces you have already met, in the order the chain rule puts them.

Check it with shapes, which is faster than checking it with algebra:

X   (n, d)
w   (d,)
Xw - y      (n,)
X.T         (d, n)
X.T @ (Xw - y)   (d, n) @ (n,) -> (d,)   same shape as w. Correct.

If the result had come out shaped $(n,)$ you would have written X @ r instead of X.T @ r, and the shape would have told you before the loss ever failed to descend.

Setting this gradient to zero and solving gives the normal equations — the closed form is just this expression rearranged.

Your task:

Implement ls_gradient(X, y, w) returning as an array of shape (d,), where n is the number of rows of X.

Example Tests

A perfect fit has zero residual and therefore zero gradient

Input: {"X":[[1,0],[0,1]],"w":[1,2],"y":[1,2]}

Expected: [0,0]

Weights of zero leave the residual equal to minus the targets

Input: {"X":[[1,0],[0,1]],"w":[0,0],"y":[1,2]}

Expected: [-1,-2]

A single feature over three rows

Input: {"X":[[1],[2],[3]],"w":[1],"y":[2,4,6]}

Expected: [-9.33333]

Python

import numpy as np


def ls_gradient(X, y, w):
    """
    Gradient of the mean squared error of a linear model.

    Args:
        X: design matrix, shape (n, d)
        y: targets, shape (n,)
        w: weights, shape (d,)

    Returns:
        array of shape (d,)
    """
    # YOUR CODE HERE
    pass

Loading docs…