2
$\begingroup$

I came across this sentence when exploring a simple nearest neighbor classifier method using Euclidean distance (link):

The slightly odd thing about using the Euclidean distance to compare features is that each dimension (or feature) is compared using the same scale.

This got me thinking - if flat feature space implies that each feature contributes equally to the distance (score function), then curved feature space changes the scale between features, so that the features then contribute different amounts to the score function. For example, imagine we have a 2D feature space - a flat piece of paper - with two points, $X_1$ and $X_2$ on it, between which we wish to calculate the distance. If we then bend this into U-shape along, say, y-axis (so, no curvature introduced in y-dimension), the distances along the x-axis would be larger in the bent case than in the flat case:

enter image description here

In other words, feature x would contribute more to the score function than feature y. This sounds awfully like weighing the feature inputs with weight vectors. Does this imply, that weight vectors (and matrices) have a direct effect on curvature of feature space? Does an identity weight matrix (or a vector of all 1s) imply our feature space is flat (and curved otherwise)? Lastly, could it then be said that whenever we are training an ML model, we are in fact learning the approximate curvature of the feature space we wish to model?

$\endgroup$
2
  • $\begingroup$ When you say "I came across this sentence", it may be a good idea to provide the link to the source where you found this sentence. $\endgroup$ – nbro Feb 11 at 14:23
  • 1
    $\begingroup$ Certainly - added the link. It's a free course on AI for beginners, titled "Building AI" as part of "Elements of AI" website. Although one might have to register in order to view the contents, I am not sure. $\endgroup$ – Nejc Kejzar Feb 11 at 14:54
1
$\begingroup$

No, it does not take into account the curvature. But, if curvature is important for you, then, it would be a good idea to look at Ricci flow and its applications in neural networks.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.