A decision tree is a supervised machine-learning model that predicts a class or a numeric value by following a sequence of feature-based questions. Its branching, if-then structure is easy to inspect, but an unrestricted tree can overfit and small changes in its training data can produce a very different model.
What is a decision tree?
A decision tree learns rules from labeled examples. Each internal node tests a feature, each branch represents an outcome of that test, and each leaf supplies the prediction. For classification, a leaf predicts a class; for regression, it predicts a numeric value. Because the model does not assume a fixed mathematical relationship between features and outcomes, it is described as non-parametric.
As an Amazon Associate I earn from qualifying purchases.
For example, a classifier deciding whether an email is suspicious might first ask whether it contains a link, then whether the sender is known. A path through those questions ends at a leaf such as “suspicious” or “not suspicious.” Real trees choose their questions from the available training data rather than relying on rules written in advance.
How does a decision tree choose a split?
Training builds the tree recursively. At a node, the algorithm considers candidate feature tests and scores the resulting child groups. It chooses the split with the lowest weighted impurity or loss, applies it, and repeats within the children until a stopping condition is reached. In a binary threshold split, one child receives observations where feature j is at most threshold t; the other receives the remaining observations.
#1 Best Overall
This process is greedy: each node takes the best split available at that point. It does not search all possible complete trees to guarantee the globally best tree. That makes tree construction practical, but a locally attractive split can shape the rest of the model in ways that are not globally optimal. Scikit-learn’s decision-tree documentation describes this split-based approach and notes that its implementation uses an optimized version of CART.
A small classification example
Suppose a node contains 10 examples: 6 belong to class A and 4 to class B. Its class proportions are 0.6 and 0.4. A candidate question might divide the examples according to whether a feature exceeds a threshold. If the resulting child groups each contain mostly one class, that split has reduced impurity. The algorithm compares this weighted child impurity with that of other candidate splits and selects the best-scoring option under its chosen criterion.
Rank #2
What are Gini impurity and entropy?
For classification, Gini impurity and entropy are common ways to quantify how mixed the classes are at a node. A node containing only one class has zero impurity under either measure; a more evenly mixed node has greater impurity. The tree seeks splits that reduce the weighted impurity of the child nodes. Entropy-based split selection is often described in terms of information gain: the reduction in entropy after the split.
Recommended Free Tools
These are alternative criteria, not universal properties of every decision tree. The selected criterion depends on the task and implementation. For regression, a tree instead minimizes a regression loss, such as squared error between numeric targets and predictions. Scikit-learn’s documentation details the criteria available in its tree models.
Classification trees, regression trees, and algorithm families
Classification trees predict discrete labels; regression trees predict numeric values. The broad idea is the same, but the split criterion and the meaning of a leaf prediction differ. A regression tree, for instance, may choose splits that reduce squared error, while a classification tree may use Gini impurity or entropy.
Tree algorithms also vary in how they split data and handle features. ID3 is associated with categorical features and information gain; C4.5 extends the family to continuous-feature thresholds and rule conversion; C5.0 is a later proprietary Quinlan-family algorithm. CART, or Classification and Regression Trees, uses binary splits and supports both classification and regression. Scikit-learn uses an optimized CART implementation.
When comparing implementations, check the task they support, split criterion, binary or multiway branching, categorical and missing-value handling, interpretability, stability, computational cost, and controls against overfitting. No algorithm family is best for every dataset; compare candidates on data representative of the intended use.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Why can a decision tree overfit?
A tree that keeps splitting can create leaves tailored to individual training examples, including noise or unusual cases. It may then perform very well on its training data but poorly on unseen data. Trees are also high-variance models: a small change in the examples can lead to different early splits and a substantially different structure.
Best Value
Limit growth and evaluate generalization rather than relying on training accuracy alone. In scikit-learn, useful controls include:
max_depth: cap how many levels the tree can grow.min_samples_split: require a minimum number of observations in a node before splitting it.min_samples_leaf: require a minimum number of observations in each leaf.ccp_alpha: apply minimal cost-complexity pruning, which removes branches when the resulting reduction in complexity justifies the associated fit trade-off.
Choose these settings using validation data or cross-validation, then assess the final model on separate test data. Report a metric suited to the task—such as an appropriate classification metric or regression error metric—rather than treating training accuracy as proof of performance. Scikit-learn’s tree guide documents these complexity controls. IBM also explains pruning as removing less useful branches to reduce tree complexity and overfitting: IBM’s decision-tree overview.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How to start with a decision tree in scikit-learn
- Choose the task and metric. Use a classifier for class labels or a regressor for numeric targets, and decide how performance should be measured before tuning.
- Split the data for evaluation. Keep test data separate, or use cross-validation during model selection, so you do not judge generalization on examples used to fit the model.
- Fit a shallow tree first. Start with a constrained model that is easier to inspect. Set an initial
max_depthand consider minimum split and leaf sizes. - Visualize and inspect it. Check the questions at the root and subsequent nodes, the leaf predictions, and whether the paths make sense for the problem.
- Tune complexity on validation results. Adjust depth and leaf-size controls; consider
ccp_alphafor post-pruning. Increase complexity only when validation performance supports it. - Evaluate once on held-out test data. Use the task-appropriate metric and interpret the result as performance on that evaluation setup, not a guarantee for every future dataset.
Interpret feature importance cautiously
Impurity-based feature importance summarizes how much a feature reduced impurity across the fitted tree. It can be misleading when a feature has many possible split points or when the tree is overfit. Check explanations against held-out data; permutation importance can be useful where appropriate, because it tests how model performance changes when a feature’s values are shuffled.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesWhen should you choose a tree or a random forest?
A single decision tree is a good choice when you want a compact sequence of rules that can be inspected, communicated, or used to surface nonlinear patterns and feature interactions. Trees generally need less feature scaling than many other model types, and they handle both classification and regression.
A random forest combines predictions from many trees. This generally improves robustness compared with relying on one potentially unstable tree, but the combined model is less compact and harder to explain as one set of rules. If interpretability is central, a shallow tree may be easier to present; if stability and predictive performance matter more, compare a forest or another ensemble against the single tree using the same validation procedure and metric.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
