XGBoost Explained: Why Trees Still Beat Deep Learning

"Why do tree-based models still outperform deep learning on typical tabular data?" - Grinsztajn, Oyallon and Varoquaux (NeurIPS 2022, paper title)

The model making the most money at most companies is not a neural network. It is a few hundred small decision trees voting.

This week we're talking about XGBoost, the gradient boosting library behind a huge share of production fraud, credit, ranking and churn models, and yes, this can get you a very comfortable job.

Here's the idea: XGBoost is a gradient boosting library that builds decision trees one at a time, with each new tree fitting the errors the previous trees left behind. It scores every candidate split with the first and second derivatives of the loss, and penalizes large leaf values, which makes it fast, accurate and very hard to beat on tabular data.

The track record is not subtle. When Chen and Guestrin published the XGBoost paper at KDD 2016, they reported that every one of the top-10 teams in KDD Cup 2015 had used it. Six years later a NeurIPS benchmark across 45 tabular datasets found tree-based models like XGBoost still beat deep learning on medium-sized data of around 10,000 rows, even before counting how much faster they train. The first serious challenger is recent: TabPFN, a transformer pretrained on synthetic tables, was published in Nature in 2025 and beats tuned boosting on small datasets of up to about 10,000 samples. So the honest 2026 answer to "what should I use on this table?" is still boosting first, and a foundation model worth benchmarking when data is small.

How does XGBoost work?

Start with a constant prediction. For every training row, compute how the loss would change if you nudged that row's prediction: the gradient g and the curvature h (the second derivative). Grow a small tree whose splits group rows with similar gradients, set each leaf to the value that lowers the loss most, shrink it by a learning rate, add the tree to the ensemble, and repeat a few hundred times. A split is scored by how much it would reduce the loss: $$\text{Gain}=\frac{1}{2}\left[\frac{G_L^2}{H_L+\lambda }+\frac{G_R^2}{H_R+\lambda }-\frac{(G_L+G_R{)}^2}{H_L+H_R+\lambda }\right]-\gamma$$ G and H are the sums of gradients and curvatures in each child, lambda shrinks leaf values toward zero and gamma is the price of adding a leaf. If the gain does not beat gamma, the split is not made. That is XGBoost's built-in pruning.

Does XGBoost use gradient descent?

Yes, but in function space rather than weight space. Each new tree is a step along the negative gradient of the loss with respect to the current predictions. XGBoost goes one better than plain gradient boosting and uses the curvature too, so each leaf takes a Newton step: its value is minus the sum of gradients divided by the sum of curvatures plus lambda.

Is XGBoost a neural network?

No. XGBoost is an ensemble of decision trees. Nothing is learned by backpropagation, the model is not differentiable with respect to its inputs, and its splits are axis-aligned thresholds on individual features. That is exactly why it handles messy tables with skewed, mixed-type and missing features so well.

Now I'm going to explain what this means for you:

The Skill Employers EXPECT

Learn to use gradient boosting well. XGBoost interview questions come up in almost every applied ML loop, and you should be able to:

• Explain boosting versus bagging in terms of bias and variance
• Name what learning_rate, max_depth, n_estimators, subsample, colsample_bytree and min_child_weight each do
• Tune with early stopping on a validation set instead of guessing the number of trees
• Train an XGBoost classifier in Python on a real table and beat a logistic regression baseline
• Read feature importance critically, and know that gain-based importance and SHAP values can disagree

If you cannot say why lowering the learning rate usually means adding trees, you are tuning by superstition.

The Skill That Separates You

Learn the objective, not just the API.

Senior engineers bend boosting to the business instead of the other way round. You should know how to:

• Derive the gradient and curvature of a loss, and plug a custom objective into XGBoost
• Compute a leaf weight and a split gain by hand from those two numbers
• Add monotone constraints so a credit model cannot lower risk as debt goes up
• Explain how XGBoost routes missing values, and when that learned default is a leak
• Know when to stop boosting and benchmark a tabular foundation model on small data instead

The Project To Learn These Skills This Week

Replicate the XGBoost paper from scratch in NumPy, with no xgboost and no scikit-learn ensembles. This is a graded Pro project on GRADuateML: XGBoost's objective, from the paper. You implement Chen and Guestrin's second-order objective and exact greedy split finding, then boost it into a real model scored on 1,000 hidden rows. Not on Pro? The free challenge below is the gain formula from above, by hand.

Requirements (each one is a graded milestone):

• Derive the gradient and curvature of logistic loss and match them to 5 decimal places.
• Compute the optimal leaf weight in closed form.
• Implement exact greedy split finding on the second-order gain, with lambda = 1 and max depth 3, and match the reference root split exactly.
• Fit one boosting round and reach ROC-AUC of at least 0.74.
• Boost 60 rounds with a learning rate of 0.3 and reach ROC-AUC of at least 0.83 with log loss of 0.50 or lower.
• Write up the replication: what the curvature term buys you that plain gradient boosting does not.

This project teaches a practical lesson for production ML:

Once you can write the objective, every hyperparameter stops being a knob and becomes a term in an equation.

If you want to sharpen your machine learning skills even more, I also selected a free challenge problem for you this week. It is XGBoost's leaf weight and split gain, by hand:

If you learned something from this newsletter, make sure to forward it to a friend.

GRADuateML

The full XGBoost paper replication, graded against hidden data, is part of GRADuateML Pro.