Data Science Decision Tree

How Decision Trees Make Predictions

A decision tree is built from a root node, a series of internal decision nodes, and leaf nodes that hold the final prediction. Starting at the root, every internal node tests a single feature against a threshold, for example 'is petal length less than 2.5 cm?', and routes the example left or right depending on the answer. Following the path from root to leaf for a given example produces its predicted class or value.

The tree is built by picking, at each node, the single feature and threshold that best separates the examples currently at that node into groups that are as homogeneous as possible with respect to the target. This process repeats recursively on each resulting group, growing the tree branch by branch, until a stopping condition is reached, such as a maximum depth or a minimum number of samples per leaf.

Measuring Split Quality: Gini and Entropy

To compare candidate splits, the algorithm needs a number that captures how mixed the classes are in a group; this is called impurity. Gini impurity estimates the probability of misclassifying a randomly chosen example if you labeled it randomly according to the group's class proportions; it is 0 when a group contains only one class and higher as the mix becomes more even. Entropy, borrowed from information theory, measures the same idea in bits: it is 0 for a pure group and increases as the class distribution becomes more uncertain. Both metrics agree on which splits are better; they mainly differ in how sharply they penalize impure groups.

• A Gini impurity or entropy of 0 means a node is perfectly pure, containing only one class.
• The algorithm evaluates many candidate (feature, threshold) pairs and picks the one that reduces impurity the most across the resulting child nodes.
• This impurity reduction is sometimes called information gain when measured with entropy.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

data = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.25, random_state=0
)

clf = DecisionTreeClassifier(criterion='gini', max_depth=3, random_state=0)
clf.fit(X_train, y_train)

print('Feature importances:', clf.feature_importances_.round(2))
print('Test accuracy:', round(clf.score(X_test, y_test), 3))

In the table above, splitting on petal length drives the impurity all the way down to zero, meaning it perfectly separates one class from the rest, so the tree-building algorithm would choose it over the sepal width split, which leaves the groups much more mixed.

Left unrestricted, a decision tree can keep splitting until every leaf contains a single training example, which perfectly fits the training data but rarely generalizes well. This is the same overfitting problem discussed for train/test splits, and it is especially easy for trees to fall into because they have no built-in resistance to memorizing noise.

• Pros: easy to interpret, handles both numeric and categorical features, no scaling required.
• Cons: prone to overfitting when grown too deep, and can be unstable since small data changes may produce a very different tree.
• Ensemble methods like random forests and gradient boosting build many trees to address these weaknesses.