I've been studying NNs with tensorflow and decided to code a simple NN from scratch to get a better idea on hwo they work.

It my understanding that the cost is used in backpropagation, so basically you calculate the error between prediction and actual and backpropagate from there.

However, in all the examples I read online, even the ones doing classification, just use:

error=actual-prediction instead of: error=mse(actual-prediction) or: error=cross_entropy(actual-prediction)

And they leave mae/rmse etc just as a metric, as per my understanding (probably wrong) understanding, these should/could be used to calculate the error as well. On the other hand, while working with tensorflow, the loss function I use, does change the output and its not just a metric.

What's my error in here?

In other words, isn't the loss function same as the error function?

Example code (taken from: machinelearninggeek.com/backpropagation-neural-network-using-python/):

Note how the MSE is used as metric only, while backpropagation only uses pred-outputs. (E1 = A2 - y_train)

for itr in range(iterations):

# feedforward propagation
# on hidden layer
Z1 = np.dot(x_train, W1)
A1 = sigmoid(Z1)

# on output layer
Z2 = np.dot(A1, W2)
A2 = sigmoid(Z2)

# Calculating error
mse = mean_squared_error(A2, y_train)
acc = accuracy(A2, y_train)
results=results.append({"mse":mse, "accuracy":acc},ignore_index=True )

# backpropagation
E1 = A2 - y_train
dW1 = E1 * A2 * (1 - A2)

E2 = np.dot(dW1, W2.T)
dW2 = E2 * A1 * (1 - A1)

W2_update = np.dot(A1.T, dW1) / N
W1_update = np.dot(x_train.T, dW2) / N

W2 = W2 - learning_rate * W2_update
W1 = W1 - learning_rate * W1_update
$$$$


I think you are confused about the terms error, loss function and metric.

To summarize:

1. Error (vector) tells us how off each prediction is.
2. Loss function maps the error (vector) to a single number (loss).
3. Loss and metric (scalar) describes overall how well the predictions are, serving as direction towards which the model should improve.

Note these terms are general and do not apply only to neural networks.

TL;DR

Let's take a regression problem as example. Suppose we have a regression model which predicts [-1, 3, 2, 4] where the ground truth is [0, 1, 2, 3]. Clearly, some predictions are off. But by how far? Intuitively, we can get the "off" part by subtraction:

prediction - actual = [-4, 3, 0, 7] - [0, 1, 2, 3] = [-4, 2, 2, 4]

which is what we call the error (vector).

Now we show the error to the model and say, "hey, those are the wrongs you made, do better next round", to which the model replies, "sure, which one do you prefer?" and hands you 2 new predictions:

1. [-2, 3, 0, 5]
2. [0, 1, 2, 10]

Which one is better? Well, it depends on you goal: if you want "as close to the truth as possible on average", then 1) may be better; or if you say "no prediction should be below the truth", then 2) is the only choice. Regardless, you have to tell the model which one direction it should improve towards - and this is what loss function is for.

Let's say we decide to use mean-squared error (MSE) as loss function. Computing gives:

Loss_1 = Mean(Square(error vector)) = Mean(Square([-2,3,0,5] - [0,1,2,3])) = Mean(Square([-2,2,-2,2])) = Mean([4,4,4,4]) = 4

Loss_2 = Mean(Square(error vector)) = Mean(Square([0,1,2,10] - [0,1,2,3])) = Mean(Square([0,0,0,7])) = Mean([0,0,0,49]) = 12.25

We call the result of loss function loss (scalar). We want the loss to be small (minimize), so we tell the model, "1) is the better way to go". Notice that the loss is a scalar, and is an indicator of goodness of prediction. In real, a model improves its prediction ('learn') by repeating

make predictions -> compute error vector -> compute loss -> feed to optimization algorithm -> update model parameters -> make new predictions

(Often you would see the terms loss and metric used interchangeably; in brief, metric is the ultimate goal you want to achieve, a number you want to maximize; however for practical reason the optimization process prefers differentiable function, but metric may not be. In this case we usually choose a loss function which is both differentiable and 'close' to the metric. Since MSE is differentiable, it can serve as loss and metric.)

Update

In the code quoted, the computation of loss is hidden in the line dW1 = E1 * A2 * (1 - A2) (and similarly in the upcoming E2 and dW2 lines). Here, back-propagation is applied which implicitly assumes a squared loss is used. Through some maths tricks and simplification, we arrive at the E1 formula. You may find the full derivation here.

• Thank you so much for your clear response, now it makes a lot more sense. So the error vector (prediction-actual) is always the same regardless of a regression and classification?. As for the loss function, this is used on the optimization algorithm (such as adam), is this correct? Jan 27 at 4:16
• 1) for the error vector - I think yes, at least no exception comes to my head. 2) yes, loss (function) is for the optimization algorithm. Jan 27 at 4:20
• by chance is there any online resource you can suggest to read more on the subject? every numpy neural network I can find uses the error vector as you described but I can't find any example of what they do with the loss function itself, other than just calculating it. Jan 27 at 4:53
• do you mean on e.g. why we pick one loss function other than another? Jan 27 at 6:27
• No, I mean where is exactly the loss function used?, all the codes I find, they calculate the error vector and use it on the backpropagation. Not the loss. Jan 27 at 13:39

There are different functions to pick from for your loss. The most common read online is the Mean Squared Error.

It calculates how far the prediction is off from the actual value and squares it. All of them are summed up.

So basically the Loss-Function "generalizes" the error over every prediction. The actual error in a case is just how far off it is.

• In my code I have: code error=outputs-training_set_outputs #derivation calcs with error code` So by your answer I understand that the MSE would be used as a metric of how well the model generalizes only? Jan 26 at 14:39
• The loss function is a way to evaluate your whole model, yes. However I don't get what your error should be. For me it looks like your error is just your output Jan 26 at 14:44
• if I use different loss functions in tensorflow, I do get different results, is this because the loss function is used in the optimizer? (rather than changing the error function error=y-y_hat) Jan 26 at 15:04
• The answer above is really good. However it's natural that different Loss Functions give you different results. You can calculate it by hand if you want, the Mean Squared Error is also often taught in school. It's totally doable. Just think of a small example, choose different functions and then calculate. You'll see easily why you'll get different results. Just keep in mind that having a smaller error is not always good. You generally want to avoid Over- and Underfitting. Also it depends on what you want to do. There is not just one solution in AI Jan 27 at 9:48