In multilayer perceptron neural networks, are the names "delta", "gradient" and "error" all the same thing? or not?

Question

What is the difference between "delta", "gradient" and "error", are these names the same thing?

I'm confused because someone once told me that both the names "delta" and "error" are commonly used. And I searched the internet, and some explanations also use the name "gradient".

For example, this article: https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/, that teaches you how to create a multilayer perceptron neural network in Python, uses the name "delta", but don't call it "gradient".

But, this other article: https://home.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html, it does not use the name "delta" or "gradient".

Please, could someone help me understand about "error", "delta" and "gradient", what is the difference between these names? are they the same thing?

Answer 1

The terms "error", "delta" and "gradient" in neural network back-propagation are often used as shorthand, or loose explanations for the same thing. This is not strictly correct in all cases, and you should be careful to check the code in context. However, these words do make for suitable variable names, so you will see them in implenentations.

Back-propagation is repeated application of the chain rule of calculus, which can be written:

$$\frac{d}{dx} f(g(x)) = \frac{df}{dg} \frac{dg}{dx}$$

This is useful for neural networks, because each layer (and each activation function) is applied as a function to each previous layer, so a neural network is effectively a giant composite of much simpler functions, each applied to the output of the previous one.

The goal of back-propagation is to figure out the gradient of the loss function with respect to the neural network's changeable parameters (the weights and biases usually). So "gradient" is kind of shorthand for this, although you work it out in part, plus some are temporary useful values that are not part of the final array of partial derivatives.

Some people like to think of the gradient as an "error correction signal", and also with careful design, the initial gradient of the loss for the neuron output with respect to the output layer, before the activation is applied, can be $\hat{y} - y$, the difference between the current output and the ground truth. So it even looks a little bit like an error value, even though it is a different kind of quantity which happens to equal the error value.

I am not sure, but I think "delta" is a direct reference to the Greek letter delta in the partial derivative $\frac{\delta L}{\delta W_i}$, as the full gradient is an array of these partial derivatives of the loss function with respect to each parameter in turn.

A good way to understand and isolate yourself from all the different names used in implementation code is to learn the mathematical representation of backpropagation. The maths gives you a less ambiguous view of what is going on, and doesn't use any of those three terms.