Data Science Grid Search

Why Hyperparameters Need Tuning

Most machine learning models have hyperparameters, settings chosen before training rather than learned from the data, such as the regularization strength C in logistic regression, the number of neighbors k in KNN, or the max_depth of a decision tree. Different values of these hyperparameters can change model performance substantially, and the best value is rarely obvious in advance. Grid search removes the guesswork by testing a defined grid of candidate values and measuring performance for each one.

How Exhaustive Search Works

Grid search is 'exhaustive' because it evaluates every possible combination of the hyperparameter values you specify, rather than sampling randomly or using an optimization shortcut. If you specify 4 values for one hyperparameter and 3 values for another, grid search trains and evaluates 4 x 3 = 12 combinations in total. For each combination, it does not just fit once: it uses cross-validation, splitting the training data into folds, fitting on some folds and validating on the held-out fold, and repeating so every fold gets a turn as the validation set. The reported score for each combination is the average across folds, which makes the comparison far more reliable than a single train/test split.

• Define the model you want to tune (e.g. LogisticRegression, KNeighborsClassifier).
• Define a parameter grid: a dictionary mapping each hyperparameter name to a list of candidate values.
• Choose the number of cross-validation folds (cv) and a scoring metric.
• GridSearchCV fits the model once per combination per fold, then reports the average score per combination.
• The combination with the best average cross-validation score is refit on the entire training set automatically.

from sklearn.datasets import load_iris
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import GridSearchCV, train_test_split

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=1)

param_grid = {
    'n_neighbors': [1, 3, 5, 7, 9],
    'weights': ['uniform', 'distance']
}

grid = GridSearchCV(
    KNeighborsClassifier(),
    param_grid=param_grid,
    cv=5,
    scoring='accuracy'
)
grid.fit(X_train, y_train)

print('Best params:', grid.best_params_)
print('Best CV score:', grid.best_score_)
print('Test accuracy:', grid.score(X_test, y_test))

Reading the Results

After fitting, a GridSearchCV object exposes best_params_ (the winning combination), best_score_ (its average cross-validation score), and best_estimator_ (a model already refit on the full training data using those params). The full grid of results, including the score and timing for every single combination, is available in the cv_results_ attribute as a dictionary that converts easily into a pandas DataFrame for inspection.

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

param_grid = {'C': [0.01, 0.1, 1, 10, 100]}

grid = GridSearchCV(
    LogisticRegression(max_iter=5000),
    param_grid=param_grid,
    cv=5
)
grid.fit(X, y)

print('Best C:', grid.best_params_['C'])
print('Best score:', round(grid.best_score_, 3))