The basic (and usual) algorithm used to update the weights of the artificial neural network (ANN) is an iterative, [numerical][3] and [optimization algorithm][4], called [_gradient descent_][2], which is based on and requires the computation of the derivative of the function you want to find the minimum of. If the function you want to find the minimum of is [multivariable][5], then, rather than the derivative, gradient descent requires the [gradient][1], which is a vector where the $i$th element contains the _partial_ derivative of the function with respect to the $i$th variable. Hence the name _gradient descent_, where the derivative of a function of one variable can be considered the gradient of the function. In the case of ANNs, we usually have a loss function that we want to minimize: for example, the mean squared error (MSE). Therefore, in order to apply gradient descent to find the minimum of the MSE, we need to find the derivative or, more precisely, the gradient of the MSE. To do it, the back-propagation (an algorithm based on the [chain rule][6]) is often used, given that the MSE is a function of the ANN, which is a composite function of multiple non-linear functions, the activation functions, whose main purpose is thus to introduce non-linearity, or, in other words, it makes the ANN powerful. Given that the MSE is a function of the parameters of the ANN, then we need to find the partial derivative of the MSE with respect to all parameters of the ANN. In this process, we will also need to find the derivatives of the activation functions that each neuron applies to its linear combination of weights: _to fully see this, you will need to learn the details of back-propagation!_ Hence the importance of the derivatives of the activation functions. A constant derivative would always give the same _learning signal_, independently of the error, but this is not desirable. _To fully understand all these statements, I recommend you learn about back-propagation and gradient descent in detail, which requires a little bit of effort!_ [1]: https://en.wikipedia.org/wiki/Gradient [2]: https://en.wikipedia.org/wiki/Gradient_descent [3]: https://en.wikipedia.org/wiki/Numerical_analysis [4]: https://en.wikipedia.org/wiki/Mathematical_optimization [5]: https://en.wikipedia.org/wiki/Multivariable_calculus [6]: https://en.wikipedia.org/wiki/Chain_rule [7]: https://en.wikipedia.org/wiki/Rectifier_(neural_networks)