Return to Answer

The basic (and usual) algorithm used to update the weights of the artificial neural network (ANN) is an iterative, numerical and optimization algorithm, called gradient descent, which is based on and requires the computation of the derivative of the function you want to find the minimum of. If the function you want to find the minimum of is multivariable, then, rather than the derivative, gradient descent requires the gradient, which is a vector where the $i$th element contains the partial derivative of the function with respect to the $i$th variable. Hence the name gradient descent, where the derivative of a function of one variable can be considered the gradient of the function.

In the case of ANNs, we usually have a loss function that we want to minimize: for example, the mean squared error (MSE). Therefore, in order to apply gradient descent to find the minimum of the MSE, we need to find the derivative or, more precisely, the gradient of the MSE. To do it, the back-propagation (an algorithm based on the chain rule) is often used. Why do we need this fancy algorithm called back-propagation? Essentially, the MSE is a function of the ANN, which is a composite function of multiple non-linear functions: the activation functions (the main purpose of the use of activation function is thus to introduce non-linearity, that is, in other words, it makes the ANN powerful). Given that the MSE is a function of the parameters of the ANN, then we need to find the partial derivative of the MSE with respect to all parameters of the ANN. In this process, we will also need to find the derivatives of the activation functions that each neuron applies to its linear combination of weights. Hence the importance of the derivatives of the activation functions.

A constant derivative would always give the same learning signal, independently of the error, but this is, of course, not desirable.

To fully understand all these statements, I recommend you learn about back-propagation and gradient descent in detail, which requires a little bit of effort!

The basic (and usual) algorithm used to update the weights of the artificial neural network (ANN) is an iterative, numerical and optimization algorithm, called gradient descent, which is based on and requires the computation of the derivative of the function you want to find the minimum of. If the function you want to find the minimum of is multivariable, then, rather than the derivative, gradient descent requires the gradient, which is a vector where the $i$th element contains the partial derivative of the function with respect to the $i$th variable. Hence the name gradient descent, where the derivative of a function of one variable can be considered the gradient of the function.

In the case of ANNs, we usually have a loss function that we want to minimize: for example, the mean squared error (MSE). Therefore, in order to apply gradient descent to find the minimum of the MSE, we need to find the derivative or, more precisely, the gradient of the MSE. To do it, the back-propagation (an algorithm based on the chain rule) is often used, given that the MSE is a function of the ANN, which is a composite function of multiple non-linear functions, the activation functions, whose main purpose is thus to introduce non-linearity, or, in other words, it makes the ANN powerful. Given that the MSE is a function of the parameters of the ANN, then we need to find the partial derivative of the MSE with respect to all parameters of the ANN. In this process, we will also need to find the derivatives of the activation functions that each neuron applies to its linear combination of weights: to fully see this, you will need to learn the details of back-propagation! Hence the importance of the derivatives of the activation functions.

A constant derivative would always give the same learning signal, independently of the error, but this is not desirable.

To fully understand all these statements, I recommend you learn about back-propagation and gradient descent in detail, which requires a little bit of effort!

The basic (and usual) algorithm used to update the weights of the artificial neural network (ANN) is an iterative, numerical and optimization algorithm, called gradient descent, which is based on and requires the computation of the derivative of the function you want to find the minimum of. If the function you want to find the minimum of is multivariable, then, rather than the derivative, gradient descent requires the gradient, which is a vector where the $i$th element contains the partial derivative of the function with respect to the $i$th variable. Hence the name gradient descent, where the derivative of a function of one variable can be considered the gradient of the function.

In the case of ANNs, we usually have a loss function that we want to minimize: for example, the mean squared error (MSE). Therefore, in order to apply gradient descent to find the minimum of the MSE, we need to find the derivative or, more precisely, the gradient of the MSE. To do it, the back-propagation (an algorithm based on the chain rule) is often used. Why do we need this fancy algorithm called back-propagation? Essentially, the MSE is a function of the ANN, which is a composite function of multiple non-linear functions: the activation functions (the main purpose of the use of activation function is thus to introduce non-linearity, that is, in other words, it makes the ANN powerful). Given that the MSE is a function of the parameters of the ANN, then we need to find the partial derivative of the MSE with respect to all parameters of the ANN. In this process, we will also need to find the derivatives of the activation functions that each neuron applies to its linear combination of weights. Hence the importance of the derivatives of the activation functions. (If it is still not clear, I suggest you learn about back-propagation.)

A constant derivative would always give the same learning signal, independently of the error, but this is, of course, not desirable.

To fully understand all these statements, I recommend you learn about back-propagation and gradient descent in detail, which requires a little bit of effort!

The basic (and usual) algorithm used to update the weights of the artificial neural network (ANN) is an iterative, numerical and optimization algorithm, called gradient descent, which is based on and requires the computation of the derivative of the function you want to find the minimum of. If the function you want to find the minimum of is multivariable, then, rather than the derivative, gradient descent requires the gradient, which is a vector where the $i$th element contains the partial derivative of the function with respect to the $i$th variable. Hence the name gradient descent, where the derivative of a function of one variable can be considered the gradient of the function.

In the case of ANNs, we usually have a loss function that we want to minimize: for example, the mean squared error (MSE). Therefore, in order to apply gradient descent to find the minimum of the MSE, we need to find the derivative or, more precisely, the gradient of the MSE. To do it, the back-propagation (an algorithm based on the chain rule) is often used. Why do we need this fancy algorithm called back-propagation? Essentially, the MSE is a function of the ANN, which is a composite function of multiple non-linear functions: the activation functions (the main purpose of the use of activation function is thus to introduce non-linearity, that is, in other words, it makes the ANN powerful). Given that the MSE is a function of the parameters of the ANN, then we need to find the partial derivative of the MSE with respect to all parameters of the ANN. In this process, we will also need to find the derivatives of the activation functions that each neuron applies to its linear combination of weights. Hence the importance of the derivatives of the activation functions.

A constant derivative would always give the same learning signal, independently of the error, but this is, of course, not desirable.

To fully understand all these statements, I recommend you learn about back-propagation and gradient descent in detail, which requires a little bit of effort!

The basic (and usual) algorithm used to update the weights of the artificial neural network (ANN) is an iterative, numerical and optimization algorithm, called gradient descent, which is based on and requires the computation of the derivative of the function you want to find the minimum of. If the function you want to find the minimum of is multivariable, then, rather than the derivative, gradient descent requires the gradient, which is a vector where the $i$th element contains the partial derivative of the function with respect to the $i$th variable. Hence the name gradient descent, where the derivative of a function of one variable can be considered the gradient of the function.

In the case of ANNs, we usually have a loss function that we want to minimize: for example, the mean squared error (MSE). Therefore, in order to apply gradient descent to find the minimum of the MSE, we need to find the derivative or, more precisely, the gradient of the MSE. To do it, the back-propagation (an algorithm based on the chain rule) is often used. Why do we need this fancy algorithm called back-propagation? Essentially, the MSE is a function of the ANN, which is a composite function of multiple non-linear functions: the activation functions (the main purpose of the use of activation function is thus to introduce non-linearity, that is, in other words, it makes the ANN powerful). Given that the MSE is a function of the parameters of the ANN, then we need to find the partial derivative of the MSE with respect to all parameters of the ANN. In this process, we will also need to find the derivatives of the activation functions that each neuron applies to its linear combination of weights. Hence the importance of the derivatives of the activation functions. (If it is still not clear, I suggest you learn about back-propagation.)

A constant derivative would always give the same learning signal, independently of the error, but this is, of course, not desirable.

To fully understand all these statements, I recommend you learn about back-propagation in detail!

The basic (and usual) algorithm used to update the weights of the artificial neural network (ANN) is an iterative, numerical and optimization algorithm, called gradient descent, which is based on and requires the computation of the derivative of the function you want to find the minimum of. If the function you want to find the minimum of is multivariable, then, rather than the derivative, gradient descent requires the gradient, which is a vector where the $i$th element contains the partial derivative of the function with respect to the $i$th variable. Hence the name gradient descent, where the derivative of a function of one variable can be considered the gradient of the function.

In the case of ANNs, we usually have a loss function that we want to minimize: for example, the mean squared error (MSE). Therefore, in order to apply gradient descent to find the minimum of the MSE, we need to find the derivative or, more precisely, the gradient of the MSE. To do it, the back-propagation (an algorithm based on the chain rule) is often used. Why do we need this fancy algorithm called back-propagation? Essentially, the MSE is a function of the ANN, which is a composite function of multiple non-linear functions: the activation functions (the main purpose of the use of activation function is thus to introduce non-linearity, that is, in other words, it makes the ANN powerful). Given that the MSE is a function of the parameters of the ANN, then we need to find the partial derivative of the MSE with respect to all parameters of the ANN. In this process, we will also need to find the derivatives of the activation functions that each neuron applies to its linear combination of weights. Hence the importance of the derivatives of the activation functions. (If it is still not clear, I suggest you learn about back-propagation.)

A constant derivative would always give the same learning signal, independently of the error, but this is, of course, not desirable.

To fully understand all these statements, I recommend you learn about back-propagation and gradient descent in detail, which requires a little bit of effort!