3
$\begingroup$

Depending on the source, I find people using different variations of the "squared error function". How come that be?

Here, it is defined as

$$ E_{\text {total }}=\sum \frac{1}{2}(\text {target}-\text {output})^{2} $$

OTOH, here, it's defined as

$$ \frac{1}{m} \sum_{i=1}^{m}\left(h_{\theta}\left(x^{(i)}\right)-y^{(i)}\right)^{2} $$

Notice that it is being divided by 1 over $m$ as opposed to variation 1, where we multiply by $1/2$.

The stuff inside the $()^2$ is simply notation, I get that, but dividing by $1/m$ and $1/2$ will clearly get a different result. Which version is the "correct" one, or is there no such thing as a correct or "official" squared error function?

$\endgroup$
0

1 Answer 1

3
$\begingroup$

The first variation is named "$E_{total}$". It contains a sum which is not very well-specified (has no index, no limits). Rewriting it using the notation of the second variation would lead to:

$$E_{total} = \sum_{i = 1}^m \frac{1}{2} \left( y^{(i)} - h_{\theta}(x^{(i)}) \right)^2,$$

where:

  • $x^{(i)}$ denotes the $i$th training example
  • $h_{\theta}(x^{(i)})$ denotes the model's output for that instance/example
  • $y^{(i)}$ denotes the ground truth / target / label for that instance
  • $m$ denotes the number of training examples

Because the term inside the large brackets is squared, the sign doesn't matter, so we can rewrite it (switch around the subtracted terms) to:

$$E_{total} = \sum_{i = 1}^m \frac{1}{2} \left( h_{\theta}(x^{(i)}) - y^{(i)} \right)^2.$$


Now it already looks quite a lot like your second variation.

The second variation does still have a $\frac{1}{m}$ terms outside the sum. That is because your second variation computes the mean squared error over all the training examples, rather than the total error computed by the first variation.

Either error can be used for training. I'd personally lean towards using the mean error rather than the total error, mainly because the scale of the mean error is independent of the batch size $m$, whereas the scale of the total error is proportional to the batch size used for training. Either option is valid, but they'll likely require different hyperparameter values (especially for the learning rate), due to the difference in scale.


With that $\frac{1}{m}$ term explained, the only remaining difference is the $\frac{1}{2}$ term inside the sum (can also be pulled out of the sum), which is present in the first variation but not in the second. The reason for including that term is given in the page you linked to for the first variation:

The $\frac{1}{2}$ is included so that exponent is cancelled when we differentiate later on. The result is eventually multiplied by a learning rate anyway so it doesn’t matter that we introduce a constant here.

$\endgroup$
2
  • $\begingroup$ question: Notice in variation 1 he just uses a sigma symbol without specifying what to sum, can he do that? Don't he have to denote the start and end value? e.g. "start: i=0" and "end: m" as in the second variation. $\endgroup$ Sep 22, 2018 at 20:34
  • $\begingroup$ @SebastianNielsen I don't think it's entirely correct to do it like that mathematically. It's not uncommon though, when it is relatively "obvious" what we're summing over (e.g. summing over all training examples), it can be convenient to simply leave the indices out. In my opinion, nothing should be considered "obvious" though in what seems to be very much an introductory-level tutorial thing. $\endgroup$
    – Dennis Soemers
    Sep 23, 2018 at 8:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .