# What is loss function in Neural Networks?

I've been studying NNs with tensorflow and decided to code a simple NN from scratch to get a better idea on hwo they work.

It my understanding that the cost is used in backpropagation, so basically you calculate the error between prediction and actual and backpropagate from there.

However, in all the examples I read online, even the ones doing classification, just use:

error=actual-prediction instead of: error=mse(actual-prediction) or: error=cross_entropy(actual-prediction)

And they leave mae/rmse etc just as a metric, as per my understanding (probably wrong) understanding, these should/could be used to calculate the error as well. On the other hand, while working with tensorflow, the loss function I use, does change the output and its not just a metric.

What's my error in here?

In other words, isn't the loss function same as the error function?

Example code (taken from: machinelearninggeek.com/backpropagation-neural-network-using-python/):

Note how the MSE is used as metric only, while backpropagation only uses pred-outputs. (E1 = A2 - y_train)

for itr in range(iterations):

# feedforward propagation
# on hidden layer
Z1 = np.dot(x_train, W1)
A1 = sigmoid(Z1)

# on output layer
Z2 = np.dot(A1, W2)
A2 = sigmoid(Z2)

# Calculating error
mse = mean_squared_error(A2, y_train)
acc = accuracy(A2, y_train)
results=results.append({"mse":mse, "accuracy":acc},ignore_index=True )

# backpropagation
E1 = A2 - y_train
dW1 = E1 * A2 * (1 - A2)

E2 = np.dot(dW1, W2.T)
dW2 = E2 * A1 * (1 - A1)

W2_update = np.dot(A1.T, dW1) / N
W1_update = np.dot(x_train.T, dW2) / N

W2 = W2 - learning_rate * W2_update
W1 = W1 - learning_rate * W1_update
$$$$


I think you are confused about the terms error, loss function and metric.

To summarize:

1. Error (vector) tells us how off each prediction is.
2. Loss function maps the error (vector) to a single number (loss).
3. Loss and metric (scalar) describes overall how well the predictions are, serving as direction towards which the model should improve.

Note these terms are general and do not apply only to neural networks.

TL;DR

Let's take a regression problem as example. Suppose we have a regression model which predicts [-4, 3, 0, 7] where the ground truth is [0, 1, 2, 3]. Clearly, some predictions are off. But by how far? Intuitively, we can get the "off" part by subtraction:

prediction - actual = [-4, 3, 0, 7] - [0, 1, 2, 3] = [-4, 2, 2, 4]

which is what we call the error (vector).

Now we show the error to the model and say, "hey, those are the wrongs you made, do better next round", to which the model replies, "sure, which one do you prefer?" and hands you 2 new predictions:

1. [-2, 3, 0, 5]
2. [0, 1, 2, 10]

Which one is better? Well, it depends on you goal: if you want "as close to the truth as possible on average", then 1) may be better; or if you say "no prediction should be below the truth", then 2) is the only choice. Regardless, you have to tell the model which one direction it should improve towards - and this is what loss function is for.

Let's say we decide to use mean-squared error (MSE) as loss function. Computing gives:

Loss_1 = Mean(Square(error vector)) = Mean(Square([-2,3,0,5] - [0,1,2,3])) = Mean(Square([-2,2,-2,2])) = Mean([4,4,4,4]) = 4

Loss_2 = Mean(Square(error vector)) = Mean(Square([0,1,2,10] - [0,1,2,3])) = Mean(Square([0,0,0,7])) = Mean([0,0,0,49]) = 12.25

We call the result of loss function loss (scalar). We want the loss to be small (minimize), so we tell the model, "1) is the better way to go". Notice that the loss is a scalar, and is an indicator of goodness of prediction. In real, a model improves its prediction ('learn') by repeating

make predictions -> compute error vector -> compute loss -> feed to optimization algorithm -> update model parameters -> make new predictions

(Often you would see the terms loss and metric used interchangeably; in brief, metric is the ultimate goal you want to achieve, a number you want to maximize; however for practical reason the optimization process prefers differentiable function, but metric may not be. In this case we usually choose a loss function which is both differentiable and 'close' to the metric. Since MSE is differentiable, it can serve as loss and metric.)

Update

In the code quoted, the computation of loss is hidden in the line dW1 = E1 * A2 * (1 - A2) (and similarly in the upcoming E2 and dW2 lines). Here, back-propagation is applied which implicitly assumes a squared loss is used. Through some maths tricks and simplification, we arrive at the E1 formula. You may find the full derivation here.

• Thank you so much for your clear response, now it makes a lot more sense. So the error vector (prediction-actual) is always the same regardless of a regression and classification?. As for the loss function, this is used on the optimization algorithm (such as adam), is this correct? Commented Jan 27, 2023 at 4:16
• 1) for the error vector - I think yes, at least no exception comes to my head. 2) yes, loss (function) is for the optimization algorithm. Commented Jan 27, 2023 at 4:20
• by chance is there any online resource you can suggest to read more on the subject? every numpy neural network I can find uses the error vector as you described but I can't find any example of what they do with the loss function itself, other than just calculating it. Commented Jan 27, 2023 at 4:53
• do you mean on e.g. why we pick one loss function other than another? Commented Jan 27, 2023 at 6:27
• No, I mean where is exactly the loss function used?, all the codes I find, they calculate the error vector and use it on the backpropagation. Not the loss. Commented Jan 27, 2023 at 13:39

There are different functions to pick from for your loss. The most common read online is the Mean Squared Error.

It calculates how far the prediction is off from the actual value and squares it. All of them are summed up.

So basically the Loss-Function "generalizes" the error over every prediction. The actual error in a case is just how far off it is.

• In my code I have: code error=outputs-training_set_outputs #derivation calcs with error code So by your answer I understand that the MSE would be used as a metric of how well the model generalizes only? Commented Jan 26, 2023 at 14:39
• The loss function is a way to evaluate your whole model, yes. However I don't get what your error should be. For me it looks like your error is just your output Commented Jan 26, 2023 at 14:44
• if I use different loss functions in tensorflow, I do get different results, is this because the loss function is used in the optimizer? (rather than changing the error function error=y-y_hat) Commented Jan 26, 2023 at 15:04
• The answer above is really good. However it's natural that different Loss Functions give you different results. You can calculate it by hand if you want, the Mean Squared Error is also often taught in school. It's totally doable. Just think of a small example, choose different functions and then calculate. You'll see easily why you'll get different results. Just keep in mind that having a smaller error is not always good. You generally want to avoid Over- and Underfitting. Also it depends on what you want to do. There is not just one solution in AI Commented Jan 27, 2023 at 9:48

A loss function is the function which can get the losses(differences) between a model's predictions and true values to evaluate how good a model is. *Loss function is also called Cost Function or Error Function.

There are popular loss function as shown below:

(1) L1 Loss:

• can compute the average of the sum of the absolute losses(differences) between a model's predictions and true values.
• 's formula is as shown below:
• 's pros are as shown below:
• Less sensitive to outliers.
• We can easily complare the losses because they are just made absolute so the range of them is not big.
• 's cons are as shown below:
• is used for a regression model.
• is also called Mean Absolute Error(MAE).
• is L1Loss() in PyTorch.

(2) L2 Loss:

• can compute the average of the sum of the squared losses(differences) between a model's predictions and true values.
• 's formula is as shown below:
• 's pros are as shown below:
• All squared losses can be differentiable.
• 's cons are as shown below:
• Sensitive to outliers.
• We cannot easily complare the losses because they are squared so the range of them is big.
• is used for a regression model.
• is also called Mean Squared Error(MSE).
• is MSELoss() in PyTorch

(3) Huber Loss:

• can do the similar computation of either L1 Loss or L2 Loss depending on the absolute losses(differences) between a model's predictions and true values compared with delta which you set. *Memos:
• delta is 1.0 basically.
• Be careful, the computation is not exactly same as L1 Loss or L2 Loss according to the formulas below.
• 's formula is as shown below. *The 1st one is L2 Loss-like one and the 2nd one is L1 Loss-like one:
• 's pros are as shown below:
• Less sensitive to outliers.
• All losses can be differentiable.
• We can more easily complare the losses than L2 Loss because only small losses are squared so the range of them is smaller than L2 Loss.
• 's cons are as shown below:
• The computation is more than L1 Loss and L2 Loss because the formula is more complex than them.
• is used for a regression model.
• is HuberLoss() in PyTorch.
• with delta` of 1.0 is same as Smooth L1 Loss which is SmoothL1Loss() in PyTorch.

(4) BCE(Binary Cross Entropy) Loss:

• can compute the differences between a model's binary predictions and true binary values.
• s' formula is as shown below:
• is used for Binary Classification. *Binary Classification is the technology to classify data into two classes.
• is also called Binary Cross Entropy or Log(Logarithmic) Loss.
• is BCELoss() in PyTorch. *Memos:

(5) Cross Entropy Loss:

• can compute the differences(0~1) between a model's predictions and true values.
• s' formula is as shown below:
• is used for Multiclass Classification and Computer Vision. *Memos:
• Multiclass Classification is the technology to classify data into multiple classes.
• Computer vision is the technology which enables a computer to understand objects.
• is CrossEntropyLoss() in PyTorch.