Please help me understand the role of loss function in neural networks

Question

I've been studying NNs with tensorflow and decided to code a simple NN from scratch to get a better idea on hwo they work.

It my understanding that the cost is used in backpropagation, so basically you calculate the error between prediction and actual and backpropagate from there.

However, in all the examples I read online, even the ones doing classification, just use:

error=actual-prediction instead of: error=mse(actual-prediction) or: error=cross_entropy(actual-prediction)

And they leave mae/rmse etc just as a metric, as per my understanding (probably wrong) understanding, these should/could be used to calculate the error as well. On the other hand, while working with tensorflow, the loss function I use, does change the output and its not just a metric.

What's my error in here?

In other words, isn't the loss function same as the error function?

Example code (taken from: machinelearninggeek.com/backpropagation-neural-network-using-python/):

Note how the MSE is used as metric only, while backpropagation only uses pred-outputs. (E1 = A2 - y_train)

for itr in range(iterations):    
    
    # feedforward propagation
    # on hidden layer
    Z1 = np.dot(x_train, W1)
    A1 = sigmoid(Z1)

    # on output layer
    Z2 = np.dot(A1, W2)
    A2 = sigmoid(Z2)
    
    
    # Calculating error
    mse = mean_squared_error(A2, y_train)
    acc = accuracy(A2, y_train)
    results=results.append({"mse":mse, "accuracy":acc},ignore_index=True )
    
    # backpropagation
    E1 = A2 - y_train
    dW1 = E1 * A2 * (1 - A2)

    E2 = np.dot(dW1, W2.T)
    dW2 = E2 * A1 * (1 - A1)

    
    # weight updates
    W2_update = np.dot(A1.T, dW1) / N
    W1_update = np.dot(x_train.T, dW2) / N

    W2 = W2 - learning_rate * W2_update
    W1 = W1 - learning_rate * W1_update
```

Answer 1

I think you are confused about the terms error, loss function and metric.

To summarize:

1. Error (vector) tells us how off each prediction is.
2. Loss function maps the error (vector) to a single number (loss).
3. Loss and metric (scalar) describes overall how well the predictions are, serving as direction towards which the model should improve.

Note these terms are general and do not apply only to neural networks.

TL;DR

Let's take a regression problem as example. Suppose we have a regression model which predicts [-1, 3, 2, 4] where the ground truth is [0, 1, 2, 3]. Clearly, some predictions are off. But by how far? Intuitively, we can get the "off" part by subtraction:

prediction - actual = [-4, 3, 0, 7] - [0, 1, 2, 3] = [-4, 2, 2, 4]

which is what we call the error (vector).

Now we show the error to the model and say, "hey, those are the wrongs you made, do better next round", to which the model replies, "sure, which one do you prefer?" and hands you 2 new predictions:

1. [-2, 3, 0, 5]
2. [0, 1, 2, 10]

Which one is better? Well, it depends on you goal: if you want "as close to the truth as possible on average", then 1) may be better; or if you say "no prediction should be below the truth", then 2) is the only choice. Regardless, you have to tell the model which one direction it should improve towards - and this is what loss function is for.

Let's say we decide to use mean-squared error (MSE) as loss function. Computing gives:

Loss_1 = Mean(Square(error vector)) = Mean(Square([-2,3,0,5] - [0,1,2,3])) = Mean(Square([-2,2,-2,2])) = Mean([4,4,4,4]) = 4

Loss_2 = Mean(Square(error vector)) = Mean(Square([0,1,2,10] - [0,1,2,3])) = Mean(Square([0,0,0,7])) = Mean([0,0,0,49]) = 12.25

We call the result of loss function loss (scalar). We want the loss to be small (minimize), so we tell the model, "1) is the better way to go". Notice that the loss is a scalar, and is an indicator of goodness of prediction. In real, a model improves its prediction ('learn') by repeating

make predictions -> compute error vector -> compute loss -> feed to optimization algorithm -> update model parameters -> make new predictions

(Often you would see the terms loss and metric used interchangeably; in brief, metric is the ultimate goal you want to achieve, a number you want to maximize; however for practical reason the optimization process prefers differentiable function, but metric may not be. In this case we usually choose a loss function which is both differentiable and 'close' to the metric. Since MSE is differentiable, it can serve as loss and metric.)

Update

In the code quoted, the computation of loss is hidden in the line dW1 = E1 * A2 * (1 - A2) (and similarly in the upcoming E2 and dW2 lines). Here, back-propagation is applied which implicitly assumes a squared loss is used. Through some maths tricks and simplification, we arrive at the E1 formula. You may find the full derivation here.

Answer 2

There are different functions to pick from for your loss. The most common read online is the Mean Squared Error.

It calculates how far the prediction is off from the actual value and squares it. All of them are summed up.

So basically the Loss-Function "generalizes" the error over every prediction. The actual error in a case is just how far off it is.