I came across this sentence when exploring a simple nearest neighbor classifier method using Euclidean distance (link):
The slightly odd thing about using the Euclidean distance to compare features is that each dimension (or feature) is compared using the same scale.
This got me thinking - if flat feature space implies that each feature contributes equally to the distance (score function), then curved feature space changes the scale between features, so that the features then contribute different amounts to the score function. For example, imagine we have a 2D feature space - a flat piece of paper - with two points, $X_1$ and $X_2$ on it, between which we wish to calculate the distance. If we then bend this into U-shape along, say, y-axis (so, no curvature introduced in y-dimension), the distances along the x-axis would be larger in the bent case than in the flat case:
In other words, feature x would contribute more to the score function than feature y. This sounds awfully like weighing the feature inputs with weight vectors. Does this imply, that weight vectors (and matrices) have a direct effect on curvature of feature space? Does an identity weight matrix (or a vector of all 1s) imply our feature space is flat (and curved otherwise)? Lastly, could it then be said that whenever we are training an ML model, we are in fact learning the approximate curvature of the feature space we wish to model?
