How do I interpret this loss function?

Question

In this AI note from https://deeplearning.ai, the loss function below is used for a regression problem. However, I don't know how to interpret this loss function.

First, does the author take the square of the difference between y-hat (prediction) and y (ground truth), so that positive and negative numbers don't cancel each other out? If so, why do we take the norm or distance, as well? Isn't the norm positive anyway? Or does he take the square so that it's more convenient to calculate the derivative?

Second, what does the other 2 in subscript mean? It's not explained in the note and I was also not able to derive it from the context. All I know is that y ∈ R.

It seems like I find it difficult to read mathematical notations. If you know a resource that explains these notations, please let me know.

Answer 1

This is the sum of squared residuals, and it uses notation from the mathematical subfield of linear algebra (arguably functional analysis).

The double vertical bars indicate that we use the mathematical operation of a norm, which is a special kind of measure of distance that holds familiar properties.

The subscript $2$ means that it is the $L_2$ norm. Until you know this notation, there’s no way to know that, but it’s so common that it’s worth remembering what an $L_p$ norm means. Let $x=(x_1,\dots,x_n)\in\mathbb R^n$ be a vector of real numbers.

$$ \vert \vert x \vert\vert_p = \left( \sum_{i=1}^n\vert x_i\vert^p \right)^{1/p} $$

For the $2$-norm in your equation, this is just the usual Euclidean distance.

The $2$ superscript has its usual meaning, that you square the $L_2$ norm once you calculate it.

Finally, $y-\hat y$ means that you take the vector of true $y$-values and subtract the vector of predicted $y$-values, denoted by $\hat y$.

In total:

1. Take the difference between the true and predicted values.

2. Square each component of that difference.

3. Add up all of those squared differences.

4. Take the square root of that sum.

5. Square that square root.

As the last step undoes the fourth step to get you back to the third step, you can stop at step $3$.

EDIT

It is common to refer to the residuals as errors. From a statistical standpoint, this is not quite correct, though it seems to be a reasonable slang that does not cause confusion among experienced practitioners.