A decision tree is a non-parametric supervised-learning model that predicts an outcome by asking a sequence of feature questions. CART (Classification and Regression Trees) builds the tree by testing feature-and-threshold pairs, selecting the split with the greatest impurity or loss reduction, and repeating that process on both child nodes. Classic CART produces a binary tree and supports both classification and regression.
What a decision tree is
A tree recursively partitions feature space into smaller regions. Internal nodes contain split rules, branches represent the outcomes of those rules, and leaves contain the final prediction.
As an Amazon Associate I earn from qualifying purchases.
- Classification: a leaf predicts a class, usually the majority class among training samples that reach it. Class probabilities can be estimated from the class proportions in that leaf.
- Regression: a leaf predicts a numeric value, commonly an average of the training targets in that region.
Because trees make no assumption that the relationship between a feature and target is linear, they are called non-parametric. They can represent interactions and abrupt changes, but an unrestricted tree can memorize training data.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
How prediction works
To classify or estimate a new observation, start at the root and apply its split rule. Follow the matching branch, apply the next rule, and continue until reaching a leaf. The leaf’s stored class or numeric prediction is the output.
#1 Best Overall
How CART chooses a split
At a node, CART evaluates candidate pairs consisting of a feature and a split point. For a numeric feature, a candidate might be “income ≤ 50,000.” The samples are divided into left and right child nodes, and the children receive a weighted impurity or loss score:
weighted score = (nleft/n) × scoreleft + (nright/n) × scoreright
CART selects the candidate with the lowest weighted score, equivalently the largest reduction from the parent node’s score. It then searches for another split independently in each child until a stopping rule prevents further growth.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Classification criteria
- Gini impurity: 1 − Σ pk2, where pk is the proportion of class k in the node. It is zero when every sample belongs to one class.
- Entropy (information gain): −Σ pk log(pk). A pure node has zero entropy. The split is selected by the reduction in entropy.
- Log loss: evaluates the quality of predicted class probabilities rather than only class purity. Availability and behavior depend on the library and version.
Gini and entropy often produce similar trees, but they are not mathematically identical. Choose one based on the objective and the implementation’s validation results rather than assuming one is universally superior.
Regression criteria
For regression, the node score is a regression loss. Squared error is a common choice: a split is preferred when it reduces the weighted within-node squared deviations of target values. The exact set of regression criteria depends on the library version; check the estimator reference for the version you deploy.
Binary branching
Classic CART makes a two-way decision at every internal node. A multi-category feature is therefore represented through binary tests, depending on the implementation’s encoding and categorical-feature support. Binary structure keeps the search and resulting paths consistent, while a multiway method can sometimes express a categorical split in fewer levels.
Why trees overfit
If growth continues until leaves contain very few observations, the tree can model noise, outliers, and accidental training-set patterns. Training accuracy may keep improving while validation performance worsens. Deep trees also become harder to inspect and can be unstable: a small change in the data may change an early split and restructure many descendants.
Controls that limit tree complexity
Use validation or cross-validation to select complexity settings. No single value is optimal for every data set.
max_depth
Caps the number of edges from the root to the deepest leaf. A lower value gives a smaller, easier-to-explain model; a higher value allows more interactions but raises overfitting risk.
min_samples_split
Requires a node to contain at least a specified number of samples before it can be split. Increasing it prevents very small parent nodes from generating additional rules.
min_samples_leaf
Requires every resulting leaf to contain at least a specified number of samples. This is a direct way to keep predictions from depending on one or two observations and is especially useful with noisy targets.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →max_leaf_nodes
Limits the total number of leaves, providing a direct budget for the number of terminal regions.
Rank #4
min_impurity_decrease
Permits a split only when its weighted impurity reduction reaches the configured threshold. This removes weak splits even when depth and sample-count limits would allow them.
Minimal cost-complexity pruning
Cost-complexity pruning compares a subtree’s fit with a penalty for its number of leaves. A fully grown tree can be generated, then subtrees whose additional fit is not worth their complexity can be removed. Select the pruning strength with a validation set or cross-validation, not from training accuracy alone.
Gini impurity or entropy?
Both criteria are valid classification choices. Gini uses squared class proportions and is often a convenient default; entropy is information-theoretic and can be useful when probability separation is important. The practical difference is data-dependent, and neither criterion guarantees better generalization. Compare them under the same cross-validation splits, preprocessing, and complexity controls. If probability quality matters, evaluate a proper probability metric in addition to accuracy or a confusion matrix.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteCART versus C4.5
CART and C4.5 are closely related tree-induction methods, but their design priorities differ.
| Axis | Classic CART | Classic C4.5 |
|---|---|---|
| Targets | Classification and numerical-target regression | Primarily classification; regression support is not the defining feature |
| Tree shape | Binary splits | Can use multiway splits for categorical attributes |
| Split measure | Impurity or loss reduction, such as Gini or squared error | Information-based measures, including gain-related criteria |
| Rule sets | Builds a tree; the scikit-learn guide describes CART as not computing rule sets | Associated with generating rules from trees |
| Categorical features | Support depends on the implementation and version | Traditional implementations were designed to handle categorical attributes |
| Pruning | Includes cost-complexity pruning in common implementations | Uses its own pruning approach; details vary by implementation |
| Reproducibility | Can be affected by feature-order randomization and tied split improvements | Depends on the particular implementation |
The official scikit-learn guide describes CART as “very similar to C4.5,” while highlighting numerical-target regression and the absence of rule-set generation as key differences. Treat categorical-feature claims as version-specific: scikit-learn’s 1.2 documentation stated that its tree implementation did not support categorical variables directly, so verify the exact version and estimator you use.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Using a decision tree in scikit-learn
The estimator names and accepted criteria should be checked against the scikit-learn version installed in your environment. A typical classification workflow looks like this:
- Split the data into training and validation portions, or define cross-validation folds.
- Fit
DecisionTreeClassifierwith a criterion such as"gini","entropy", or, where supported,"log_loss". - Search complexity settings such as
max_depth,min_samples_leaf, andccp_alphausing validation performance. - Refit the selected configuration on the available training data and evaluate once on untouched test data.
from sklearn.tree import DecisionTreeClassifier
model = DecisionTreeClassifier(
criterion="gini",
max_depth=5,
min_samples_leaf=10,
random_state=42,
)
model.fit(X_train, y_train)
predictions = model.predict(X_test)For a numeric target, use DecisionTreeRegressor and select a regression criterion supported by your installed version. The example values above are illustrative starting points, not universal recommendations.
Recommended Free Tools
Make fitting reproducible
The current classifier reference notes that features are randomly permuted at each split. When multiple candidate splits have the same improvement, the implementation may choose among them randomly. Set random_state when you need repeatable fitting, and record the library version, preprocessing, data split, and parameter configuration.
Strengths and limitations
Strengths
- Predictions can be explained as a sequence of explicit rules.
- They capture nonlinear effects and feature interactions without requiring a linear model specification.
- They can be used for both classification and regression in the CART framework.
- Feature scaling is generally not required for threshold-based splits.
Limitations
- Unpruned trees can overfit badly.
- Small data changes can produce a different structure.
- Very deep trees become difficult to communicate and audit.
- Class imbalance, missing values, and categorical variables require deliberate handling that varies by library.
- A single tree may be less stable than an ensemble of trees, although ensembles trade away some direct interpretability.
A practical decision checklist
- Confirm whether the target is categorical or numeric.
- Check your library version’s supported criteria, missing-value behavior, and categorical-feature support.
- Choose a baseline criterion, then compare alternatives with the same cross-validation procedure.
- Set a complexity control before interpreting training performance.
- Inspect leaf sizes, validation error, and the resulting paths for implausible rules.
- Set
random_stateand record the complete fitting configuration when reproducibility matters.
Further reading
The foundational reference is Classification and Regression Trees by Leo Breiman, Jerome Friedman, Richard Olshen, and Charles Stone, published in 1984. For implementation details, consult the documentation for the exact tree estimator and version used in your project.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
