ML Decision Tree

How a Decision Tree Makes Decisions

A decision tree predicts an outcome by asking a sequence of simple yes/no questions about the input features, such as "Is petal length less than 2.5 cm?". Each question is a node that splits the data into two branches; the tree keeps splitting until it reaches a leaf, and the leaf's majority class becomes the prediction. The tricky part is figuring out which question to ask at each node — scikit-learn's DecisionTreeClassifier answers this by testing every feature and every possible threshold, and picking the split that makes the two resulting groups as pure (homogeneous) as possible.

• criterion chooses the impurity measure used to score splits — 'gini' (default) or 'entropy'.
• max_depth caps how many questions deep the tree can go; lower values fight overfitting.
• min_samples_split sets the minimum number of samples a node must have before it is allowed to split further.
• min_samples_leaf sets the minimum number of samples allowed in a leaf, preventing the tree from carving out tiny, noisy leaves.

Gini Impurity: Scoring a Split

Gini impurity measures how mixed the classes are in a node: Gini = 1 - sum(p_i^2), where p_i is the proportion of samples belonging to class i in that node. A pure node containing only one class has a Gini of 0; a node split evenly between two classes has a Gini of 0.5. For every candidate split, scikit-learn computes the weighted average Gini of the two child nodes and chooses the split — feature and threshold — that reduces impurity the most compared to the parent node.

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0
)

tree = DecisionTreeClassifier(criterion="gini", random_state=0)
tree.fit(X_train, y_train)

print("Test accuracy:", tree.score(X_test, y_test))
print("Feature importances:", tree.feature_importances_.round(3))

Controlling Depth to Avoid Overfitting

from sklearn.metrics import accuracy_score

full_tree = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)
shallow_tree = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X_train, y_train)

for name, model in [("Full tree", full_tree), ("max_depth=2", shallow_tree)]:
    train_acc = accuracy_score(y_train, model.predict(X_train))
    test_acc = accuracy_score(y_test, model.predict(X_test))
    print(f"{name}: train={train_acc:.3f} test={test_acc:.3f}")

from sklearn.tree import export_text

rules = export_text(shallow_tree, feature_names=list(load_iris().feature_names))
print(rules)