Breiman, Random Forests, Machine Learning 45(1) 2001
The target here is built from interactions. Two features multiply, two more agree in sign, a third pair fires only in one corner of the space. No single axis-aligned split captures any of that, so a shallow tree underfits badly and a deep one memorises the noise around the signal.
Breiman's argument is that you can have both: grow deep trees that fit the structure, and average enough of them that their individual mistakes cancel. Two ingredients make the averaging work.
Bootstrap sampling. Each tree sees a different resample of the rows, drawn with replacement.
Random subspaces. At every split, each tree may only choose among a random subset of m features out of p. This is what decorrelates the trees. Without it they all find the same dominant split first and averaging barely helps.
The rows a tree did not see, about of them, give you an error estimate for free with no held-out split at all.
Find the strongest single split in the whole dataset.
Gini impurity for a binary node with positive rate p is:
The gain from a split is the parent's impurity minus the size-weighted impurity of the children:
Implement explore_forest(train_df) returning n_rows, n_features, positive_rate, root_gini, best_feature and best_gain.
Search every feature f0 to f7, and for each one the thresholds np.unique(np.quantile(col, np.linspace(0.1, 0.9, 12))), splitting on col <= t. Skip any split leaving fewer than 5 rows on a side. Round the rates and gains to 5 places.
Notice how small the best gain is. That is the interaction structure telling you a single split cannot do much here.
sklearn.tree, no sklearn.ensemblenp.random.default_rng(1)Evaluated server-side against a hidden test set.