I read that functions are used as activation functions only when they are differentiable. What about the unit step activation function? So, is there any other reason a function can be used as an activation function (apart from being differentiable)?


Not completely sure your question. Do you mean

Q. why should we use activation function?

Ans: we need to introduce non-linearity to the network. Otherwise, multiple layers are no difference from single layer network. (It is obvious as we write things in matrix form, and say when we have two layers with weights $W_1$ and $W_2$, the two layer is no difference from a single layer with weight $W_2 W_1$.

Q. why they need to be differentiable?

Ans: Just for sake that we can back-propagate gradients back to earlier layers. Note that back-propagation is nothing but the chain rule in calculus. Say $f(\cdot)$ is an activation function in one layer and the output of that activation function is $\bf y$ and the input is ${\bf u}=W \bf x$, where $\bf x$ is output from the previous layer and mix with weights $W$ in current layer. Of course, the final loss $L$ will depend on ${\bf y} = f({\bf u})= f(W {\bf x})$. Say, loss $L=g(\bf y)$ somehow. To train the weights $W$, we have to find the gradient $\frac{\partial L}{\partial W}$ so that we can adjust weight $W$ to minimize $L$. But $\frac{\partial L}{\partial W}=\frac{\partial g({\bf y})}{\partial W}=\frac{\partial g({\bf y})}{\partial \bf y}\frac{\partial {\bf y}}{\partial {\bf u}}\frac{\partial {\bf u}}{\partial W}$. Each of these product terms can be computed locally and will be accumulatively multiplied as we apply backprop. And note that the middle term $\frac{\partial {\bf y}}{\partial {\bf u}}=\frac{\partial f({\bf u})}{\partial {\bf u}}$ is just "derivative" of $f(\cdot)$, thus we require the activation function to be differentiable and "informative"/non-zero (at least most of the time). Note that ReLU is not differentiable everywhere and that is why researchers (at least Yoshua Bengio) worried about that when they first tried to adopt ReLU. You may check out the interview of Bengio by Andrew Ng for that.

Q. why step function is a bad activation function?

Ans: Note that step function is differentiable almost everywhere but is not "informative" though. For places (flat region) where it is differentiable, the derivative is simply zero. Consequently, any later layer gradient (information) will get cut off as it passes through a step function activation function.

| improve this answer | |
  • 1
    $\begingroup$ I think you should have asked for clarifications before answering. I have edited the question and I tried to clarify it, but I am not sure this is what the OP was asking. He/she will eventually clarify it. $\endgroup$ – nbro Apr 5 '19 at 19:29
  • $\begingroup$ Note that ReLU and some other non-continuously differentiable functions do possess well-defined subgradients over the full range. We can simply select one of those to use as the gradient at the non differentiable points and backprop will work fine. $\endgroup$ – Ray Apr 7 '19 at 23:31

Differentiability for activation functions is desirable but not necessary since you can reformulate the derivatives at the non-differentiable points, like in the case of ReLU.

Properties Needed:

  • To take advantage of Universal Approximation Theorem, and to take advantage of the modeling capacity it promises, the activation functions need to be continuous or Borel measurable (if this term is confusing just think of this as common functions), discriminatory (similarly discriminatory means the function doesn't just produce 0 all the time when integrating w.r.t. another function) and nonpolynomial (nonlinear).
  • Also with the recent research works it is beneficial to have it monotonically increasing.

Recent theoretical analysis of the activation functions gives the edge to ReLU with some preliminary theoretical guarantees like this "Convergence Analysis of Two-layer Neural Networks with ReLU Activation" https://arxiv.org/abs/1705.09886.

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.