Data Science K-nearest Neighbor

A Model With No Training Phase

Unlike logistic regression, KNN does not fit coefficients or learn an internal formula during training. Instead, it simply stores the entire training dataset. All the real work happens at prediction time: when a new point arrives, KNN measures its distance to every stored training point, finds the k closest ones, and lets those neighbors vote on the label. This makes KNN a lazy learner, or instance-based learner, since it defers computation until it actually needs to make a prediction.

Classifying by Majority Vote

To classify a new point: compute the distance (commonly Euclidean distance) from the new point to every point in the training set, sort those distances, and take the k nearest. Each of those k neighbors casts one vote for its own class label, and the new point is assigned whichever class received the most votes. For example, with k=5, if 3 neighbors are labeled 'A' and 2 are labeled 'B', the new point is classified as 'A'.

• Choose a value of k (the number of neighbors to consult).
• Compute the distance from the new point to every training point.
• Select the k training points with the smallest distance.
• Take a majority vote among those k neighbors' labels.
• Assign the majority label to the new point.

from sklearn.datasets import load_iris
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0)

# KNN relies on distance, so scaling features first matters
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

model = KNeighborsClassifier(n_neighbors=5)
model.fit(X_train_scaled, y_train)

print('Test accuracy:', model.score(X_test_scaled, y_test))
print('Prediction for first test row:', model.predict(X_test_scaled[:1]))

Choosing k

A small k (like k=1) makes the model sensitive to noise: a single mislabeled or unusual neighbor can flip the prediction, leading to high variance and overfitting. A large k smooths predictions out by consulting many neighbors, but too large a k can blur real local patterns and pull in points from a different class entirely, leading to underfitting. The right k is typically found using cross validation, trying several odd values of k (odd values help avoid tied votes in binary classification) and picking the one with the best average validation score.