ML K-nearest Neighbors

How KNN Classification Works

K-nearest neighbors (KNN) is often called a lazy learner because it does no real work during training beyond storing the training data. All the computation happens at prediction time: to classify a new point, KNN measures its distance to every stored training point, finds the k closest ones, and assigns the new point the majority class among those k neighbors. There is no explicit model being fit; the training data itself is the model.

Because KNN relies directly on distance calculations, the choice of distance metric and the scale of the features both matter enormously. Euclidean distance is the default and works well for continuous features on comparable scales, but a feature measured in the thousands (like salary) will dominate a feature measured in single digits (like years of experience) unless the data is scaled first. Standardizing every feature to zero mean and unit variance before fitting KNN is close to mandatory.

from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=400, n_features=6, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)

scaler = StandardScaler().fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)

knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train_scaled, y_train)

print('Test accuracy:', knn.score(X_test_scaled, y_test))

Choosing the Right Value of k

The number of neighbors, k, is the main hyperparameter to tune, and it controls a direct bias-variance tradeoff. A small k (like 1 or 3) makes the decision boundary follow individual training points closely, which can fit noise in the data and lead to high variance. A large k smooths the boundary by averaging over more neighbors, which reduces variance but can wash out real structure and push predictions toward the overall majority class, increasing bias. The best k is usually found empirically rather than guessed.

from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=400, n_features=6, random_state=2)

for k in [1, 3, 5, 7, 9, 15, 25]:
    model = KNeighborsClassifier(n_neighbors=k)
    scores = cross_val_score(model, X, y, cv=5)
    print(f'k={k:2d}  mean accuracy={scores.mean():.3f}')

• Always scale features (for example with StandardScaler) before fitting KNN, since it depends directly on distance calculations
• Use an odd value of k for binary classification so votes cannot tie
• Small k fits noise and has high variance; large k oversmooths the boundary and has high bias
• KNN has almost no training cost but a slow, memory-heavy prediction step, since every prediction scans the stored training points

from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=400, n_features=6, random_state=4)

param_grid = {'n_neighbors': [1, 3, 5, 7, 9, 11, 15]}
search = GridSearchCV(KNeighborsClassifier(), param_grid, cv=5, scoring='accuracy')
search.fit(X, y)

print('Best k:', search.best_params_['n_neighbors'])
print('Best CV accuracy:', round(search.best_score_, 3))