1
$\begingroup$

The formula for the sum of squared errors (SSE) is:

$$ \frac{1}{2} \sum_{i=1}^n (t^i - o^i)^2 $$

I have a few related questions.

  1. If $t^i - o^i$ is negative, doesn't the power of 2 eliminate any negative result?

  2. How can then exist any negative gradient at the output?

  3. What if the output is too high, and so the gradient should be negative, meaning that the weights (on average) have to be decreased? How can it be differentiated between 'output too high' and 'output too low'?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

If $t^i - o^i$ is negative, doesn't the power of 2 eliminate any negative result?

In the loss function, yes that is correct, and is what you want - a measurement that gets higher due to any difference between the predicted and correct results. Minimising the value of that measurement is a goal for the optimiser.

How can then exist any negative gradient at the output?

The gradient of the cost function is different to the cost function. Technically it is the derivative of the cost function with respect to the parameters you can change. Taking the derivative changes the powers in the equation.

If you plot the graph from a single example, you can clearly see the gradient is negative when the prediction is too low, and positive when the prediction is too high:

SSE loss function example graph

Note this is the opposite relationship to the one you suggest in the question. That is why it is called gradient descent when we apply corrections. The steps to correct the parameters are in the opposite direction to the gradient.

In terms of algebra (and moving the example index into a subscript so that is doesn't get in the way visually), assuming $o_i$ is your output value, and $t_i$ is the target value, then you want the gradient of your loss function $L$ with respect to $o_i$ or $\frac{\partial L}{\partial o_i}$:

$$\frac{\partial L}{\partial o_i} = \frac{\partial}{\partial o_i} \frac{1}{2}(t_i - o_i)^2 = \frac{\partial}{\partial o_i} \frac{1}{2}(t_i^2 - 2t_i o_i + o_i^2) = \frac{1}{2}(-2t_i + 2o_i) = o_i - t_i$$

This simple gradient value is negative if your output $o_i$ is lower than the target $t_i$, and positive if $o_i$is greater than $t_i$.

As an aside, if you wondered why the multiplier of $\frac{1}{2}$ was included in the definition of SSE, it is intended by design to end up with this simple result. Without it, everything works, just you would end up with the gradient being $2(o_i - t_i)$

Improve this answer
$\endgroup$
2
  • $\begingroup$ Nice, thanks for the clear and detailed answer. Just one more question: While the SSE is the sum of the errors of all samples, the MSE divides through the number of samples to get the mean error. But: The MSE makes far more sense in a NN, right? I mean, why should the error, and so the adjustment of the weights grow with the number of samples? $\endgroup$
    – Ben
    Commented Nov 25, 2018 at 11:49
  • $\begingroup$ @Ben: In brief, the MSE is more used in stats-based learning algorithms than SSE because it estimates the loss for a single "typical" point, and the more data you get, the better the estimate. That means you can do things such as take a small subset (mini-batch) of your full data set, measure the loss, calculate the gradient and make a step - all the while using self-consistent (but noisy) measures of the errors and gradients. Also you can make comparisons easily between different sizes of data set. If that is not clear, please ask a new separate question. $\endgroup$
    – Neil Slater
    Commented Nov 25, 2018 at 17:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .