Data Science Cross Validation
Cross validation estimates how well a model generalizes by repeatedly splitting the data into training and validation folds, so the reported performance does not depend on one lucky or unlucky split.
The Problem With a Single Train/Test Split
A single train_test_split() gives one accuracy number, but that number depends partly on which rows happened to land in the test set. Shuffle the split differently and the score can move around, especially with smaller datasets. Cross validation addresses this by testing the model on several different splits and reporting the average, giving a much more stable and trustworthy estimate of real-world performance.
How K-Fold Splitting Works
In k-fold cross validation, the dataset is divided into k equally sized folds (chunks). The model is trained k separate times: each time, one fold is held out as the validation set and the remaining k-1 folds are used for training. Every fold gets exactly one turn as the validation set, so every row in the dataset is used for validation exactly once and for training k-1 times. The final reported score is the average of the k individual validation scores.
- Split the data into k folds of roughly equal size (a common choice is k=5 or k=10).
- Round 1: train on folds 2-k, validate on fold 1.
- Round 2: train on folds 1, 3-k, validate on fold 2.
- ...continue until every fold has served as the validation set exactly once.
- Average the k validation scores into a single cross-validation score.
Example: 5-fold cross validation with cross_val_score
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
X, y = load_iris(return_X_y=True)
model = LogisticRegression(max_iter=1000)
scores = cross_val_score(model, X, y, cv=5, scoring='accuracy')
print('Score per fold:', scores)
print('Mean accuracy:', scores.mean())
print('Standard deviation:', scores.std())Stratified K-Fold for Classification
For classification problems with imbalanced classes, plain k-fold splitting can accidentally create folds where a minority class is over- or under-represented. StratifiedKFold fixes this by preserving the original class proportions in every fold. scikit-learn's cross_val_score automatically uses stratified folds for classifiers, but you can pass a StratifiedKFold object explicitly for full control.
Example: Explicit StratifiedKFold
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
X, y = load_breast_cancer(return_X_y=True)
cv = StratifiedKFold(n_splits=10, shuffle=True, random_state=1)
model = LogisticRegression(max_iter=5000)
scores = cross_val_score(model, X, y, cv=cv)
print('Mean accuracy over 10 folds:', scores.mean())
