Data Science Bootstrap Aggregation
Bootstrap aggregation, or bagging, improves model stability and accuracy by training many copies of a model on random resampled subsets of the data and combining their predictions.
The Problem Bagging Solves
A single decision tree trained on one dataset can be unstable: a small change in the training data can produce a noticeably different tree, which means high variance in its predictions. Bagging reduces that variance by training many models instead of one, each on a slightly different view of the data, and then averaging (or voting on) their outputs. The combined prediction is typically smoother and more reliable than any single model's.
Bootstrapping: Sampling With Replacement
The 'bootstrap' half of the name refers to how each model's training set is built. If the original training set has n rows, a bootstrap sample also has n rows, but they are drawn randomly with replacement from the original set. This means some original rows appear multiple times in a given bootstrap sample, while others are left out entirely (on average, about 37% of rows are excluded from any single bootstrap sample). Each of the models in the ensemble gets its own independently drawn bootstrap sample, so no two models see exactly the same training data.
- Draw a bootstrap sample (random sample with replacement, same size as the original data) for each ensemble member.
- Train an independent copy of the base model (often a decision tree) on each bootstrap sample.
- For classification, aggregate by majority vote across all the models' predictions.
- For regression, aggregate by averaging all the models' predicted values.
Example: BaggingClassifier with decision trees
from sklearn.datasets import load_breast_cancer
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import BaggingClassifier
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=42)
base_tree = DecisionTreeClassifier()
bagged_model = BaggingClassifier(
estimator=base_tree,
n_estimators=100, # train 100 trees on 100 bootstrap samples
random_state=42
)
bagged_model.fit(X_train, y_train)
print('Bagged accuracy:', bagged_model.score(X_test, y_test))
# Compare against a single tree for reference
single_tree = DecisionTreeClassifier(random_state=42)
single_tree.fit(X_train, y_train)
print('Single tree accuracy:', single_tree.score(X_test, y_test))Why Aggregation Reduces Variance
Each individual bootstrapped model tends to overfit its particular resample and makes somewhat noisy, uncorrelated errors. When you average many such models (or take a majority vote), the random errors tend to cancel out while the genuine signal, which all the models pick up on, is reinforced. This is why bagging works especially well with high-variance base learners like unpruned decision trees, and offers less benefit for already-stable, low-variance models like logistic regression.
