The basic (and usual) algorithm used to update the weights of the artificial neural network (ANN) is an iterative, [numerical][3] and [optimization algorithm][4], called [_gradient descent_][2], which is based on and requires the computation of the derivative of the function you want to find the minimum of. If the function you want to find the minimum of is [multivariable][5], then, rather than the derivative, gradient descent requires the [gradient][1], which is a vector where the $i$th element contains the _partial_ derivative of the function with respect to the $i$th variable. Hence the name _gradient descent_, where the derivative of a function of one variable can be considered the gradient of the function. 

In the case of ANNs, we usually have a loss function that we want to minimize: for example, the mean squared error (MSE). Therefore, in order to apply gradient descent to find the minimum of the MSE, we need to find the derivative or, more precisely, the gradient of the MSE. To do it, the back-propagation (an algorithm based on the [chain rule][6]) is often used. Why do we need this fancy algorithm called back-propagation? Essentially, the MSE is a function of the ANN, which is a composite function of multiple non-linear functions: the activation functions (the main purpose of the use of activation function is thus to introduce non-linearity, that is, in other words, it makes the ANN powerful). Given that the MSE is a function of the parameters of the ANN, then we need to find the partial derivative of the MSE with respect to all parameters of the ANN. In this process, we will also need to find the derivatives of the activation functions that each neuron applies to its linear combination of weights. Hence the importance of the derivatives of the activation functions. (If it is still not clear, I suggest you learn about back-propagation.)

A constant derivative is not necessarily bad for learning. Consider the [ReLU activation function][7], defined as $${\displaystyle f(x)=x^{+}=\max(0,x),}$$

whose derivative is

$$ f'(x)=
\begin{cases} 
0 & \text{if  }  x < 0 \\
1 & \text{if  }  x > 0 \\
\text{undefined} & \text{if  }  x = 0 \\
\end{cases}
$$

However, ReLU is widely used as the primary activation function of ANNs. To see this more in detail, I suggest you learn about back-propagation.


 [1]: https://en.wikipedia.org/wiki/Gradient
 [2]: https://en.wikipedia.org/wiki/Gradient_descent
 [3]: https://en.wikipedia.org/wiki/Numerical_analysis 
 [4]: https://en.wikipedia.org/wiki/Mathematical_optimization
 [5]: https://en.wikipedia.org/wiki/Multivariable_calculus
 [6]: https://en.wikipedia.org/wiki/Chain_rule
 [7]: https://en.wikipedia.org/wiki/Rectifier_(neural_networks)