Variations of the squared error function

Question

Depending on the source I find people using different variations of the "squared error function", how come that be?

Variation 1

Variation 2

Notice that it is being devided by 1 over m as opposed to variation 1 (1/2)

The stuff inside the ()^2 is simply notation I get that, but dividing by 1/m and 1/2 will cleary get a different result. Which version is the "correct" one, or is there no such thing as a correct or "official" squared error function?

Answer 1

The first variation is named "$E_{total}$". It contains a sum which is not very well-specified (has no index, no limits). Rewriting it using the notation of the second variation would lead to:

$$E_{total} = \sum_{i = 1}^m \frac{1}{2} \left( y^{(i)} - h_{\theta}(x^{(i)}) \right)^2,$$

where:

• $x^{(i)}$ denotes the $i$th training example
• $h_{\theta}(x^{(i)})$ denotes the model's output for that instance/example
• $y^{(i)}$ denotes the ground truth / target / label for that instance
• $m$ denotes the number of training examples

Because the term inside the large brackets is squared, the sign doesn't matter, so we can rewrite it (switch around the subtracted terms) to:

$$E_{total} = \sum_{i = 1}^m \frac{1}{2} \left( h_{\theta}(x^{(i)}) - y^{(i)} \right)^2.$$

---

Now it already looks quite a lot like your second variation.

The second variation does still have a $\frac{1}{m}$ terms outside the sum. That is because your second variation computes the mean squared error over all the training examples, rather than the total error computed by the first variation.

Either error can be used for training. I'd personally lean towards using the mean error rather than the total error, mainly because the scale of the mean error is independent of the batch size $m$, whereas the scale of the total error is proportional to the batch size used for training. Either option is valid, but they'll likely require different hyperparameter values (especially for the learning rate), due to the difference in scale.

---

With that $\frac{1}{m}$ term explained, the only remaining difference is the $\frac{1}{2}$ term inside the sum (can also be pulled out of the sum), which is present in the first variation but not in the second. The reason for including that term is given in the page you linked to for the first variation:

The $\frac{1}{2}$ is included so that exponent is cancelled when we differentiate later on. The result is eventually multiplied by a learning rate anyway so it doesn’t matter that we introduce a constant here.