Have a go at it. The editor and the docs panel are open, and your code is saved as you type. Running it needs a free account — you’ll come back to exactly what you wrote.
The Two-Sample t-Statistic
Model A scores 0.82, model B scores 0.85. Is B better, or did B get a friendlier test split?
The t-statistic answers how many standard errors apart are these two means:
The numerator is the difference you observed. The denominator is how big a difference you would expect from sampling noise alone. Their ratio is the whole idea: a difference is only interesting relative to the noise it sits in.
Roughly, ∣t∣>2 is the conventional threshold for "probably not noise". A large difference between two very noisy, very small samples can easily give ∣t∣<1.
A = [1, 2, 3, 4] mean 2.5 s^2 = 1.6667 n = 4 B = [3, 4, 5, 6] mean 4.5 s^2 = 1.6667 n = 4 t = (2.5 - 4.5) / sqrt(1.6667/4 + 1.6667/4) = -2.191
This is the version that does not assume the two groups share a variance (Welch's), which is the one you almost always want.
Your task:
Implement t_statistic(a, b) returning t as a float. Use the unbiased variance (ddof=1) for each group.
Example Tests
Two samples two units apart, with matching spread
Input: {"a":[1,2,3,4],"b":[3,4,5,6]}
Expected: -2.19089
Identical samples give exactly zero
Input: {"a":[1,2,3],"b":[1,2,3]}
Expected: 0
The same gap, but far noisier, is much less convincing
Input: {"a":[1,5,9,13],"b":[3,7,11,15]}
Expected: -0.54772
import numpy as np
def t_statistic(a, b):
"""
Welch's two-sample t-statistic.
Args:
a: first sample, shape (n_a,)
b: second sample, shape (n_b,)
Returns:
float: (mean_a - mean_b) / sqrt(var_a/n_a + var_b/n_b)
"""
# YOUR CODE HERE
pass