XGBoost builds an ensemble by adding decision trees one at a time, choosing each tree to improve a regularized objective. It uses both gradients and Hessians—the first and second derivatives of the loss—to score leaf weights and candidate splits. The weighted quantile sketch is a key technique from the original paper for finding approximate split candidates; it is not a description of every tree method in current XGBoost.
What XGBoost is
XGBoost is a scalable system for gradient-boosted trees introduced by Tianqi Chen and Carlos Guestrin. Their paper appeared in the proceedings of KDD 2016: “XGBoost: A Scalable Tree Boosting System”.
As an Amazon Associate I earn from qualifying purchases.
In gradient boosting, the model is built in stages. Each new tree adjusts the model’s predictions based on the current state of the ensemble and the objective being optimized. The final prediction comes from combining the contributions of the trees, rather than choosing one tree to solve the entire problem.
How the objective guides tree building
XGBoost’s boosted-tree tutorial describes the objective as the training loss plus a penalty on model complexity. The loss measures how well predictions fit the training targets; the penalty discourages unnecessarily complex trees and oversized leaf scores. This means a candidate tree is judged not just by whether it reduces training error, but by whether its improvement justifies its added complexity. See the official Introduction to Boosted Trees tutorial.
#1 Best Overall
Why use gradients and Hessians?
At a boosting step, the tutorial approximates the loss with a second-order Taylor expansion around the model’s current predictions. For each training example, that approximation uses two quantities: the gradient, or first derivative, and the Hessian, or second derivative. The gradient indicates the direction and local rate of change in loss; the Hessian captures how that rate changes locally.
Aggregating these statistics for examples assigned to a leaf gives the learner a practical way to estimate a leaf’s contribution and compare tree structures. The Hessian therefore is not simply another name for a residual: it supplies curvature information that helps determine how strongly a proposed update should affect predictions.
Leaf scores and split gain
The regularization described in the tutorial penalizes both the number of leaves and the squared values of their weights. For a fixed tree structure, the tutorial derives a closed-form solution for the leaf weights. A proposed split can then be assessed by comparing the score from the gradient and Hessian sums on its two sides with the score before splitting, while accounting for the complexity cost of adding a branch.
This is why “fit a tree to residuals” is an incomplete explanation of XGBoost. The learner evaluates candidate structures through a regularized objective built from first- and second-order loss information.
What the weighted quantile sketch contributes
A tree learner may have too many distinct feature values to evaluate every possible split efficiently. Approximate split finding reduces that search by selecting a manageable set of candidate thresholds. The original XGBoost paper identifies the weighted quantile sketch as a contribution for approximate tree learning, alongside a sparsity-aware algorithm. The authors describe it this way: “We propose a novel sparsity-aware algorithm for sparse data and weighted quantile sketch for approximate tree learning.”
Conceptually, the sketch summarizes feature values to support candidate split selection while taking instance weights associated with the objective into account. It is therefore not ordinary, unweighted quantile binning. Nor does the paper’s contribution mean that every tree-building method in current XGBoost uses this same sketch in the same way.
How the paper’s sketch relates to current tree methods
Current XGBoost documentation distinguishes tree construction methods by how they generate and evaluate split candidates. The names and descriptions below are from the project’s XGBoost Parameters reference; these are distinct approaches, not interchangeable labels for the weighted quantile sketch.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
| Method | Split-candidate approach | Documented distinction |
|---|---|---|
exact |
Enumerates split candidates. | Uses exact candidate enumeration rather than the approximate strategies described for the other methods. |
approx |
Uses a quantile sketch and gradient histogram. | Approximate split finding; the parameter reference describes its sketch-based approach. |
hist |
Uses a histogram-optimized approximate greedy algorithm. | The reference describes it as faster. The documentation says auto behaves as hist. |
The table reflects the project’s current parameter documentation, which can change over time. In particular, it is useful to keep the historical contribution and present-day method options separate: the weighted quantile sketch helps explain the paper’s approximate tree-learning design, while hist is documented as a histogram-optimized approximate method.
Why XGBoost was designed to scale
The 2016 paper describes more than a split-selection technique. Its authors identify sparsity-aware split finding and systems work involving cache access patterns, compression, and sharding as parts of the system’s scalability approach. These details help explain why XGBoost became known as a system for large-scale tree boosting, not just a new formula for choosing a tree.
The paper’s abstract says that XGBoost “scales beyond billions of examples using far fewer resources than existing systems.” That is the authors’ claim in their 2016 paper, not a workload-independent guarantee or a current benchmark result. Actual performance depends on the data, configuration, hardware, and software version.
Putting the pieces together
- Start with current predictions. The model already contains the trees added in earlier boosting steps.
- Compute local loss information. Gradients and Hessians summarize the first- and second-order behavior of the loss for training examples.
- Consider tree structures. The learner evaluates leaves and candidate splits using aggregated gradient and Hessian information, with penalties for model complexity.
- Choose a split-finding strategy. Depending on the configured tree method, candidates may be enumerated exactly or found through an approximate approach such as quantile-sketch or histogram-based methods.
- Add the selected tree. Its leaf weights update predictions, and the boosting process repeats.
In short, gradient boosting explains the staged addition of trees; the second-order objective explains how XGBoost scores tree structures and leaf updates; and the weighted quantile sketch explains one important route to approximate split finding in the original system.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallQuick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
