# Which function $(\hat{y} - y)^2$ or $(y - \hat{y})^2$ should I use to compute the gradient?

The MSE can be defined as $$(\hat{y} - y)^2$$, which should be equal to $$(y - \hat{y})^2$$, but I think their derivative is different, so I am confused of what derivative will I use for computing my gradient. Can someone explain for me what term to use?

• Wow. Do take the effort to plug a few numbers into the terms and see what the square does. Epic question – nav Jul 19 '19 at 4:17

The derivative of $$\mathcal{L_1}(y, x) = (\hat{y} - y)^2 = (f(x) - y)^2$$ with respect to $$\hat{y}$$, where $$f$$ is the model and $$\hat{y} = f(x)$$ is the output of the model, is

\begin{align} \frac{d}{d \hat{y}} \mathcal{L_1} &= \frac{d}{d \hat{y}} (\hat{y} - y)^2 \\ &= 2(\hat{y} - y) \frac{d}{d \hat{y}} (\hat{y} - y) \\ &= 2(\hat{y} - y) (1) \\ &= 2(\hat{y} - y) \end{align}

The derivative of $$\mathcal{L_2}(y, x) = (y - \hat{y})^2 = (y - f(x))^2$$ w.r.t $$\hat{y}$$ is

\begin{align} \frac{d}{d \hat{y}} \mathcal{L_2} &= \frac{d}{d \hat{y}} (y - \hat{y})^2 \\ &= 2(y -\hat{y}) \frac{d}{d \hat{y}} (y -\hat{y}) \\ &= 2(y - \hat{y})(-1)\\ &= -2(y - \hat{y})\\ &= 2(\hat{y} - y) \end{align}

So, the derivatives of $$\mathcal{L_1}$$ and $$\mathcal{L_2}$$ are the same.

The MSE can be defined as $$(\hat{y} - y)^2$$, which should be equivalent to $$(y - \hat{y})^2$$

They are not just "equivalent". It is actually the exact same function, with two different ways to write it.

$$(\hat{y} - y)^2 = (\hat{y} - y)(\hat{y} - y) = \hat{y}^2 -2\hat{y}y + y^2$$

$$(y - \hat{y})^2 = (y -\hat{y})(y - \hat{y}) = y^2 -2y\hat{y} + \hat{y}^2$$

These are exactly the same function. Not just "equivalent" or "equivalent everywhere", but actually the same function. It is therefore no surprise that any derivative is also the same - including the partial derivative with respect to $$\hat{y}$$ which is what you typically use to drive gradient descent.

The two ways of writing the function is because it is a square and thus has two factorisations. When you write it as a square you can choose which form to use for the inner term.

Which function [form] should I use to compute the gradient?

You can use either form, it does not matter. They represent the same function and have the same gradient.

• I wrote "equivalent" in the question, but the author had original written "equal", so it's my bad (even though equivalence does not exclude equality). – nbro May 31 '19 at 16:25
• 'The two ways of writing the function is because it is a square and thus has two roots. When you write it as a square you can choose which root to use for the inner term.' What does this mean? – DuttaA Jun 1 '19 at 0:54
• @DuttaA: Actually I got the term wrong. It's not "roots". I'll go look up the correct term . . . but essentially the equations $y = x^2$ and $y = (-x)^2$ follow the same pattern. I think the more accurate term is factorisation. – Neil Slater Jun 1 '19 at 8:11
• @NeilSlater I don't think factorization will be the right term. I think it will be 2 unique roots rather? Mathematically the terms used seems fuzzy since when we are talking about factorization we have to talk about the variables involved I guess? – DuttaA Jun 1 '19 at 8:27
• @DuttaA: Roots are the solutions (where the graph line crosses 0). Factorisations are the correct term when reversing expansion of e.g. quadratic equation. – Neil Slater Jun 1 '19 at 8:39

The derivative is the same as far as I understand it.

If $$y$$ is constant and $$\hat{y}$$ is the variable the result will be:
$$((\hat{y} - y)^2)' = 2(\hat{y} - y)$$
and for the other formula:
$$((y - \hat{y})^2)' = -2(y - \hat{y})$$
which is the same.

• I can see that the second equation can only be derived if you have taken the correct route for partial derivative (I commented earlier that it looked wrong - but actually I was wrong to say that). Usually tutorials don't consider that $y$ is a "constant", but that this is a partial derivative, where we only care about the gradient w.r.t. $\hat{y}$. The result is much the same, but either way it may help to show a step of expansion – Neil Slater May 31 '19 at 16:09