4
$\begingroup$

The MSE can be defined as $(\hat{y} - y)^2$, which should be equal to $(y - \hat{y})^2$, but I think their derivative is different, so I am confused of what derivative will I use for computing my gradient. Can someone explain for me what term to use?

$\endgroup$
  • $\begingroup$ Wow. Do take the effort to plug a few numbers into the terms and see what the square does. Epic question $\endgroup$ – nav Jul 19 at 4:17
6
$\begingroup$

The derivative of $\mathcal{L_1}(y, x) = (\hat{y} - y)^2 = (f(x) - y)^2$ with respect to $\hat{y}$, where $f$ is the model and $\hat{y} = f(x)$ is the output of the model, is

\begin{align} \frac{d}{d \hat{y}} \mathcal{L_1} &= \frac{d}{d \hat{y}} (\hat{y} - y)^2 \\ &= 2(\hat{y} - y) \frac{d}{d \hat{y}} (\hat{y} - y) \\ &= 2(\hat{y} - y) (1) \\ &= 2(\hat{y} - y) \end{align}

The derivative of $\mathcal{L_2}(y, x) = (y - \hat{y})^2 = (y - f(x))^2$ w.r.t $\hat{y}$ is

\begin{align} \frac{d}{d \hat{y}} \mathcal{L_2} &= \frac{d}{d \hat{y}} (y - \hat{y})^2 \\ &= 2(y -\hat{y}) \frac{d}{d \hat{y}} (y -\hat{y}) \\ &= 2(y - \hat{y})(-1)\\ &= -2(y - \hat{y})\\ &= 2(\hat{y} - y) \end{align}

So, the derivatives of $\mathcal{L_1}$ and $\mathcal{L_2}$ are the same.

$\endgroup$
7
$\begingroup$

The MSE can be defined as $(\hat{y} - y)^2$, which should be equivalent to $(y - \hat{y})^2$

They are not just "equivalent". It is actually the exact same function, with two different ways to write it.

$$(\hat{y} - y)^2 = (\hat{y} - y)(\hat{y} - y) = \hat{y}^2 -2\hat{y}y + y^2$$

$$(y - \hat{y})^2 = (y -\hat{y})(y - \hat{y}) = y^2 -2y\hat{y} + \hat{y}^2$$

These are exactly the same function. Not just "equivalent" or "equivalent everywhere", but actually the same function. It is therefore no surprise that any derivative is also the same - including the partial derivative with respect to $\hat{y}$ which is what you typically use to drive gradient descent.

The two ways of writing the function is because it is a square and thus has two factorisations. When you write it as a square you can choose which form to use for the inner term.

Which function [form] should I use to compute the gradient?

You can use either form, it does not matter. They represent the same function and have the same gradient.

$\endgroup$
  • $\begingroup$ I wrote "equivalent" in the question, but the author had original written "equal", so it's my bad (even though equivalence does not exclude equality). $\endgroup$ – nbro May 31 at 16:25
  • $\begingroup$ 'The two ways of writing the function is because it is a square and thus has two roots. When you write it as a square you can choose which root to use for the inner term.' What does this mean? $\endgroup$ – DuttaA Jun 1 at 0:54
  • $\begingroup$ @DuttaA: Actually I got the term wrong. It's not "roots". I'll go look up the correct term . . . but essentially the equations $y = x^2$ and $y = (-x)^2$ follow the same pattern. I think the more accurate term is factorisation. $\endgroup$ – Neil Slater Jun 1 at 8:11
  • $\begingroup$ @NeilSlater I don't think factorization will be the right term. I think it will be 2 unique roots rather? Mathematically the terms used seems fuzzy since when we are talking about factorization we have to talk about the variables involved I guess? $\endgroup$ – DuttaA Jun 1 at 8:27
  • $\begingroup$ @DuttaA: Roots are the solutions (where the graph line crosses 0). Factorisations are the correct term when reversing expansion of e.g. quadratic equation. $\endgroup$ – Neil Slater Jun 1 at 8:39
2
$\begingroup$

The derivative is the same as far as I understand it.

If $y$ is constant and $\hat{y}$ is the variable the result will be:
$((\hat{y} - y)^2)' = 2(\hat{y} - y)$
and for the other formula:
$((y - \hat{y})^2)' = -2(y - \hat{y})$
which is the same.

$\endgroup$
  • 1
    $\begingroup$ I can see that the second equation can only be derived if you have taken the correct route for partial derivative (I commented earlier that it looked wrong - but actually I was wrong to say that). Usually tutorials don't consider that $y$ is a "constant", but that this is a partial derivative, where we only care about the gradient w.r.t. $\hat{y}$. The result is much the same, but either way it may help to show a step of expansion $\endgroup$ – Neil Slater May 31 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.