ML Cross Validation

Why a Single Train/Test Split Isn't Enough

A single train_test_split gives one accuracy number, but that number depends heavily on which rows happened to land in the test set. Reshuffle the split with a different random_state and the score can move noticeably, especially on small or imbalanced datasets, making it hard to tell whether a change to the model actually helped or the score just moved because of which rows were held out this time.

Cross validation fixes this by splitting the data into k roughly equal parts, called folds, and running k rounds of training and evaluation. In each round, one fold is held out as the validation set while the model trains on the remaining k-1 folds; over the k rounds every row gets used for validation exactly once. Averaging the k scores gives a much more stable estimate of how the model performs on unseen data than any single split could.

import numpy as np
from sklearn.model_selection import KFold
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=200, n_features=6, random_state=0)
kf = KFold(n_splits=5, shuffle=True, random_state=0)

fold_scores = []
for train_idx, test_idx in kf.split(X):
    model = LogisticRegression(max_iter=1000)
    model.fit(X[train_idx], y[train_idx])
    fold_scores.append(model.score(X[test_idx], y[test_idx]))

print('Fold scores:', fold_scores)
print('Mean accuracy:', np.mean(fold_scores))

cross_val_score and Choosing k

Scikit-learn's cross_val_score function automates the loop shown above: hand it an unfitted estimator, the full feature matrix and labels, and a number of folds, and it returns an array of scores, one per fold. For classification tasks, prefer StratifiedKFold (which cross_val_score uses by default for classifiers) so that each fold keeps roughly the same proportion of each class as the full dataset; plain KFold can otherwise create folds with very few or zero examples of a minority class.

from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=300, n_features=8, random_state=7)

model = RandomForestClassifier(n_estimators=150, random_state=7)
scores = cross_val_score(model, X, y, cv=5, scoring='accuracy')

print('Scores per fold:', scores)
print('Mean +/- std: %.3f +/- %.3f' % (scores.mean(), scores.std()))

• Shuffle the data before splitting (shuffle=True) unless it is already in random order, or folds can end up unbalanced
• Use StratifiedKFold for classification so each fold keeps the same class proportions as the full dataset
• Fit any scaler, encoder, or imputer inside a Pipeline so it is refit separately on each fold's training data
• A larger k gives a less biased estimate but costs more compute and can increase variance between runs

from sklearn.model_selection import cross_validate, StratifiedKFold
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=300, n_features=8, weights=[0.7, 0.3], random_state=3)

pipeline = make_pipeline(StandardScaler(), SVC(kernel='rbf', C=1.0))
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=3)

results = cross_validate(pipeline, X, y, cv=cv, scoring=['accuracy', 'f1'])

print('Accuracy per fold:', results['test_accuracy'])
print('F1 per fold:      ', results['test_f1'])