ML Grid Search
Grid search automates the tedious work of trying every combination of hyperparameters so you can pick the setting that performs best on validation data.
Why Hyperparameters Need Tuning
Every machine learning model has hyperparameters, settings chosen before training rather than learned from data, such as the C regularization strength in logistic regression or the number of neighbors k in KNN. Choosing them by hand and re-running fit() over and over is slow and easy to get wrong, especially when several hyperparameters interact with each other.
GridSearchCV: Trying Every Combination
Example
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.neighbors import KNeighborsClassifier
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
param_grid = {
'n_neighbors': [3, 5, 7, 9, 11],
'weights': ['uniform', 'distance']
}
grid = GridSearchCV(KNeighborsClassifier(), param_grid, cv=5)
grid.fit(X_train, y_train)
print('Best parameters:', grid.best_params_)Reading the Results
GridSearchCV performs k-fold cross-validation for every combination in the grid, then stores the winning combination in best_params_, the corresponding fitted estimator in best_estimator_, and its cross-validated score in best_score_. Because it refits on the entire training set with the best parameters by default, refit=True, best_estimator_ is ready to call predict() on new data immediately.
Example
import pandas as pd
print('Best cross-validated accuracy:', grid.best_score_)
print('Test set accuracy:', grid.score(X_test, y_test))
results = pd.DataFrame(grid.cv_results_)
top_results = results[['param_n_neighbors', 'param_weights', 'mean_test_score']]
print(top_results.sort_values('mean_test_score', ascending=False).head())- param_grid: a dictionary mapping hyperparameter names to the list of values to try.
- cv: the number of cross-validation folds used to score each combination.
- scoring: the metric used to rank combinations, such as accuracy or f1.
- n_jobs: how many CPU cores to use in parallel; -1 uses all available cores.
- refit: whether to retrain the best combination on the full training set automatically.
Example
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
pipeline = Pipeline([
('scaler', StandardScaler()),
('svc', SVC())
])
param_grid = {
'svc__C': [0.1, 1, 10, 100],
'svc__kernel': ['linear', 'rbf']
}
grid = GridSearchCV(pipeline, param_grid, cv=5, n_jobs=-1)
grid.fit(X_train, y_train)
print('Best pipeline parameters:', grid.best_params_)Exercise: ML Grid Search
What is the main purpose of grid search in machine learning?
