ML Bootstrap Aggregation

What Is Bootstrap Aggregation?

Bootstrap aggregation (bagging) is an ensemble technique built to fight overfitting. Instead of training one model and trusting its single view of the data, bagging trains the same algorithm dozens or hundreds of times, each time on a different random resample of the training set, and then combines the results by majority vote for classification or by averaging for regression. Individual models may each overfit their own resample in a slightly different way, but those errors tend to cancel out when the predictions are pooled, leaving a combined model with much lower variance than any one of its members.

Each resample used to train an individual model is called a bootstrap sample: rows are drawn from the original dataset with replacement until the new sample is the same size as the original. Because the same row can be picked more than once, some rows appear several times in a given sample while others are skipped entirely, and on average about 63% of the original rows show up in any single bootstrap sample. The roughly 37% of rows left out are that model's 'out-of-bag' data, which turns out to be useful for validation, as covered later on this page.

import numpy as np

data = np.array([12, 15, 18, 22, 25, 30, 35, 40])
rng = np.random.default_rng(seed=42)

# Draw a bootstrap sample: same size, drawn with replacement
bootstrap_sample = rng.choice(data, size=len(data), replace=True)
print('Original data:   ', data)
print('Bootstrap sample:', bootstrap_sample)

Random Forest: Bagging Plus Feature Randomness

A Random Forest is bagging applied specifically to decision trees, with one added source of randomness: at every split, each tree is only allowed to choose from a random subset of the available features, a number controlled by the max_features parameter, rather than considering every feature. Restricting the candidate features at each split stops a handful of strong predictors from shaping every tree the same way, which decorrelates the trees further and lowers the variance of the averaged prediction beyond what plain bagging of full-feature trees can achieve.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

X, y = make_classification(n_samples=500, n_features=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

forest = RandomForestClassifier(n_estimators=200, random_state=42)
forest.fit(X_train, y_train)

predictions = forest.predict(X_test)
print('Accuracy:', accuracy_score(y_test, predictions))

• Averaging many bootstrapped models reduces variance without adding much bias
• Training is embarrassingly parallel, since every tree is built independently of the others
• Out-of-bag samples give a built-in validation estimate without needing a separate holdout set
• Ensembles of trees are more robust to noisy features and outliers than any single deep tree

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=500, n_features=10, random_state=1)

forest = RandomForestClassifier(n_estimators=300, oob_score=True, random_state=1)
forest.fit(X, y)

print('Out-of-bag score:', forest.oob_score_)