Chen & Guestrin, XGBoost: A Scalable Tree Boosting System, KDD 2016
Boosting fits trees one at a time, each correcting what the ensemble so far got wrong. Chen & Guestrin's contribution is to make "what it got wrong" precise: take a second-order Taylor expansion of the loss around the current prediction, and every quantity you need falls out in closed form.
With and the first and second derivatives of the loss at the current margin, the regularised objective for a tree with leaves j is:
Minimising over wj gives the optimal leaf weight and the objective it achieves:
And the gain from splitting a node into L and R is the drop in that objective:
This is not an impurity measure. It is a direct estimate of how much the loss will fall, which is why the Hessian appears in every denominator.
Establish the base rate every boosted model starts from.
Implement explore_boosting(train_df) returning:
n_rows, n_featurespositive_rate (5 places)base_margin: the log-odds of the base rate, log1−pp, rounded to 5 placesfeature_std_mean: the mean of the per-feature standard deviations (5 places)The features are f0 through f9. The base margin is what a zero-tree model would predict if you initialised it optimally. This project starts from F0=0 instead, which costs the first round or two.
xgboost, no sklearn.ensembleEvaluated server-side against a hidden test set.