Hyperparameter Tuning in KNN

The performance of a KNN model depends heavily on the choice of k, the number of neighbors considered - too small a k can overfit, while too large a k can oversmooth the decision boundary.

The Effect of k

Small k (e.g. k=1):
- Sensitive to noise, can overfit

Large k:
- Smoother boundary, can underfit
- Slower predictions

Tuning with GridSearchCV

from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import GridSearchCV

param_grid = {"n_neighbors": [1, 3, 5, 7, 9, 11]}

grid = GridSearchCV(KNeighborsClassifier(), param_grid, cv=5)
grid.fit(X_train, y_train)

print("Best k:", grid.best_params_)
print("Best score:", grid.best_score_)

Other Hyperparameters

Besides k, the distance metric (Euclidean, Manhattan) and the weighting scheme (uniform vs distance-based) can also be tuned to improve performance on a given dataset.

Coming Up Next

Next, you'll move on to a more powerful classifier - the Support Vector Machine.