0
$\begingroup$

If I understand right that each observation whithin a dataset, creates a different loss surface where we want to find the global minimum.

How different those surfaces one from another? Would it be correct to say that they differ like (for example) those two parabolas: f(x) = 5x^2 +4x+2 versus f(x) = 5x^2 +1x+8 which can be seen as same parabola located in another place of xy plane.

Thank you

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Let's take in consideration linear regression. You have a dataset composed by $x,y$ pairs, and you assume they are linearly related, thus you model this problem with LR: $$ y = wx+b $$ Now, you want to find the $w$ and $b$ that best describe your data, thus you set a loss function, say MSE, and you minimize it: $$ L(w, b) = \sum_{(x,y)\in D} (y - (wx+b))^2 $$

As you can see if you consider a single sample, this is a parabola. However, the fact that is a parabola, it's given by the loss function, and your model

At that point, you try to minimize the average loss (the $1/n$ is discarded because it does not effect the minimization), so say you take "the average parabola across the dataset"

However, you can clearly see that there is no relation between points in your dataset, as they are assumed to be independent, thus their loss function can be as far as they want (they are still parabolas, but very far from each other)

Improve this answer
$\endgroup$
2
  • $\begingroup$ Assuming we have a good and a representative dataset, and assuming that there is a good correlation between predictor features and the predicted feature, can we say that the global minimums of each loss surfaces should overlap and fall in same coordinates of loss surfaces? ( I mean a theoretical case where i keep backpropagating and surching for global minimum of each loss surface whithout moving to.another observation till done with the current) $\endgroup$
    – Igor
    Commented Aug 13, 2023 at 12:38
  • $\begingroup$ @Igor given a distance metric (loss), your global minima would be by definition $L=0$, which means that the requirement is that the function approximated by your model intersects all the data in your dataset... however, there are infinite functions that can pass from N points, no matter the dimensionality, and the parameters of such functions might be arbitrarily far, thus global optima (s) do not overlap each other $\endgroup$
    – Alberto
    Commented Aug 13, 2023 at 22:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .