The following is from a machine learning book.

enter image description here

enter image description here

I did not understand the explanation given in the figure caption. Could some expert make it clear? Why is the stretching class-dependent for the center case?


1 Answer 1


You provided very little context here without explicit reference, though it looks like the topic here is instance based machine learning such as KNN method using potentially beneficial scaled Euclidean distance. You need to first know why the left graph case of feature-dependent scaling is effective compared to its original unstretched classification learning in terms of their decision boundaries.

Consider a hypothetical dataset with two classes (spam and non-spam) having two features (length and ratio of number of blacklisted keywords). Obviously the original features have wildly different scales thus KNN method would provide an implicit highly nonlinear irregular decision boundary by the meshgrid in below plot around the possibly overlapping region of the feature space (code is provided at the end).enter image description here

But if you scale the feature of ratio of blacklisted keywords to be of similar range of the other email length feature, the below implicit decision boundary provided by the same KNN classifier becomes much more regular and thus effective in many respects such as variance reduction.enter image description here

This is exactly the case in your left graph where stretching the feature $x_2$ makes the classification boundary much more regular and statistically effective to predict any new data.

But if you're in the middle class-dependent scaling case, any such scaling elongation or compression obviously aren't effective compared to their original unscaled graph since any such scaling doesn't make the bottom-left possibly overlapping corner of the two classes any more clearly separable and regular. Similar case goes for the right case of correlated features since the usual linear correlation between $x_1$ and $x_2$ won't get affected at all by any scaling.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.neighbors import KNeighborsClassifier
from matplotlib.colors import ListedColormap


spam_length = np.random.uniform(150, 500, 50)
spam_blacklisted_keyword = np.random.uniform(0.2, 0.8, 50)
spams = np.column_stack((spam_length, spam_blacklisted_keyword))

nonspam_length = np.random.uniform(30, 300, 50)
nonspam_blacklisted_keyword = np.random.uniform(0, 0.3, 50)
nonspams = np.column_stack((nonspam_length, nonspam_blacklisted_keyword))

# spam_length = np.random.uniform(150, 500, 50)
# spam_blacklisted_keyword = np.random.uniform(0.2*50, 0.8*50, 50)
# spams = np.column_stack((spam_length, spam_blacklisted_keyword))

# nonspam_length = np.random.uniform(30, 300, 50)
# nonspam_blacklisted_keyword = np.random.uniform(0*50, 0.3*50, 50)
# nonspams = np.column_stack((nonspam_length, nonspam_blacklisted_keyword))

X = np.vstack((spams, nonspams))
y = np.array([0] * 50 + [1] * 50)

knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X, y)

h = .2  # Step size in the mesh
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
Z = knn.predict(np.c_[xx.ravel(), yy.ravel()])

# Plotting the decision boundary
Z = Z.reshape(xx.shape)
plt.contourf(xx, yy, Z, alpha=0.4)
plt.scatter(spams[:, 0], spams[:, 1], c='red', label='spams')
plt.scatter(nonspams[:, 0], nonspams[:, 1], c='green', label='nonspams')
plt.ylabel('# of blacklisted keywords')
plt.ylim(0, 1)
plt.title('KNN Decision Boundary')

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .