3
$\begingroup$

Depending on the source I find people using different variations of the "squared error function", how come that be?

Variation 1

enter image description here

Variation 2

enter image description here

Notice that it is being devided by 1 over m as opposed to variation 1 (1/2)

The stuff inside the ()^2 is simply notation I get that, but dividing by 1/m and 1/2 will cleary get a different result. Which version is the "correct" one, or is there no such thing as a correct or "official" squared error function?

$\endgroup$
1
  • $\begingroup$ As you go in depth of Machine Learning you'll see constants everywhere matter less and less. $\endgroup$ – DuttaA Sep 22 '18 at 19:00
3
$\begingroup$

The first variation is named "$E_{total}$". It contains a sum which is not very well-specified (has no index, no limits). Rewriting it using the notation of the second variation would lead to:

$$E_{total} = \sum_{i = 1}^m \frac{1}{2} \left( y^{(i)} - h_{\theta}(x^{(i)}) \right)^2,$$

where:

  • $x^{(i)}$ denotes the $i$th training example
  • $h_{\theta}(x^{(i)})$ denotes the model's output for that instance/example
  • $y^{(i)}$ denotes the ground truth / target / label for that instance
  • $m$ denotes the number of training examples

Because the term inside the large brackets is squared, the sign doesn't matter, so we can rewrite it (switch around the subtracted terms) to:

$$E_{total} = \sum_{i = 1}^m \frac{1}{2} \left( h_{\theta}(x^{(i)}) - y^{(i)} \right)^2.$$


Now it already looks quite a lot like your second variation.

The second variation does still have a $\frac{1}{m}$ terms outside the sum. That is because your second variation computes the mean squared error over all the training examples, rather than the total error computed by the first variation.

Either error can be used for training. I'd personally lean towards using the mean error rather than the total error, mainly because the scale of the mean error is independent of the batch size $m$, whereas the scale of the total error is proportional to the batch size used for training. Either option is valid, but they'll likely require different hyperparameter values (especially for the learning rate), due to the difference in scale.


With that $\frac{1}{m}$ term explained, the only remaining difference is the $\frac{1}{2}$ term inside the sum (can also be pulled out of the sum), which is present in the first variation but not in the second. The reason for including that term is given in the page you linked to for the first variation:

The $\frac{1}{2}$ is included so that exponent is cancelled when we differentiate later on. The result is eventually multiplied by a learning rate anyway so it doesn’t matter that we introduce a constant here.

$\endgroup$
7
  • 1
    $\begingroup$ I think the main caveat you should mention is that the 'constant' outside does not matter as learning rate will take care of it anyways...since 1/2 is also introduced for convenience, so technically it should have been 1/2m ..otherwise there is not much to answer $\endgroup$ – DuttaA Sep 22 '18 at 18:59
  • $\begingroup$ @DuttaA That's already mentioned in the final quote from the original source on the first variation, right? $\endgroup$ – Dennis Soemers Sep 22 '18 at 19:01
  • $\begingroup$ Sorry then...I didn't check the sources. $\endgroup$ – DuttaA Sep 22 '18 at 19:02
  • 1
    $\begingroup$ @DuttaA I mean the quote right at the very bottom of my answer (which I quoted from the page, but now it's also inside my answer in the form of a quote box). $\endgroup$ – Dennis Soemers Sep 22 '18 at 19:03
  • $\begingroup$ Well explained Dennis! $\endgroup$ – Sebastian Nielsen Sep 22 '18 at 20:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.