Medium

Have a go at it. The editor and the docs panel are open, and your code is saved as you type. Running it needs a free account — you’ll come back to exactly what you wrote.

Entropy of a Distribution

~12 mincode completion

Entropy

Entropy measures how uncertain a distribution is — how many yes/no questions you would need, on average, to pin down the outcome.

Measured in bits. A fair coin has $H = 1$ : one question. A four-sided fair die has $H = 2$ . A coin that always lands heads has $H = 0$ , because you already know.

The maximum for $n$ outcomes is $lo g_{2} n$ , reached when every outcome is equally likely. Any concentration of probability lowers it.

p = [0.5, 0.5]        H = 1.0    bits
p = [1.0, 0.0]        H = 0.0    bits
p = [0.25]*4          H = 2.0    bits

Two implementation notes that matter more than the formula. $lo g_{2} 0$ is , but $0 lo g 0$ should contribute 0 — the limit is zero, and a zero-probability outcome carries no uncertainty. So you must skip the zeros rather than let them poison the sum with nan. This is the same class of bug as a nan cross-entropy loss, and the same fix.

Entropy is also the quantity decision trees minimise when they choose a split, and the $H (p)$ inside cross-entropy.

Your task:

Implement entropy(probs) returning $H$ in bits as a float, treating zero probabilities as contributing zero.

Example Tests

A fair coin takes exactly one bit

Input: {"probs":[0.5,0.5]}

Expected: 1

A certain outcome carries no uncertainty, and the zero must not become nan

Input: {"probs":[1,0]}

Expected: 0

Four equally likely outcomes take two bits

Input: {"probs":[0.25,0.25,0.25,0.25]}

Expected: 2

Python

import numpy as np


def entropy(probs):
    """
    Shannon entropy in bits.

    Args:
        probs: array of probabilities, shape (n,), summing to 1.
               May contain exact zeros.

    Returns:
        float: -sum(p * log2(p)), with 0*log2(0) treated as 0
    """
    # YOUR CODE HERE
    pass

Loading docs…