2
$\begingroup$

I read that functions are used as activation functions only when they are differentiable. What about the unit step activation function? So, is there any other reason a function can be used as an activation function (apart from being differentiable)?

$\endgroup$
1
$\begingroup$

Not completely sure your question. Do you mean

Q. why should we use activation function?

Ans: we need to introduce non-linearity to the network. Otherwise, multiple layers are no difference from single layer network. (It is obvious as we write things in matrix form, and say when we have two layers with weights $W_1$ and $W_2$, the two layer is no difference from a single layer with weight $W_2 W_1$.

Q. why they need to be differentiable?

Ans: Just for sake that we can back-propagate gradients back to earlier layers. Note that back-propagation is nothing but the chain rule in calculus. Say $f(\cdot)$ is an activation function in one layer and the output of that activation function is $\bf y$ and the input is ${\bf u}=W \bf x$, where $\bf x$ is output from the previous layer and mix with weights $W$ in current layer. Of course, the final loss $L$ will depend on ${\bf y} = f({\bf u})= f(W {\bf x})$. Say, loss $L=g(\bf y)$ somehow. To train the weights $W$, we have to find the gradient $\frac{\partial L}{\partial W}$ so that we can adjust weight $W$ to minimize $L$. But $\frac{\partial L}{\partial W}=\frac{\partial g({\bf y})}{\partial W}=\frac{\partial g({\bf y})}{\partial \bf y}\frac{\partial {\bf y}}{\partial {\bf u}}\frac{\partial {\bf u}}{\partial W}$. Each of these product terms can be computed locally and will be accumulatively multiplied as we apply backprop. And note that the middle term $\frac{\partial {\bf y}}{\partial {\bf u}}=\frac{\partial f({\bf u})}{\partial {\bf u}}$ is just "derivative" of $f(\cdot)$, thus we require the activation function to be differentiable and "informative"/non-zero (at least most of the time). Note that ReLU is not differentiable everywhere and that is why researchers (at least Yoshua Bengio) worried about that when they first tried to adopt ReLU. You may check out the interview of Bengio by Andrew Ng for that.

Q. why step function is a bad activation function?

Ans: Note that step function is differentiable almost everywhere but is not "informative" though. For places (flat region) where it is differentiable, the derivative is simply zero. Consequently, any later layer gradient (information) will get cut off as it passes through a step function activation function.

$\endgroup$
  • 1
    $\begingroup$ I think you should have asked for clarifications before answering. I have edited the question and I tried to clarify it, but I am not sure this is what the OP was asking. He/she will eventually clarify it. $\endgroup$ – nbro Apr 5 at 19:29
  • $\begingroup$ Note that ReLU and some other non-continuously differentiable functions do possess well-defined subgradients over the full range. We can simply select one of those to use as the gradient at the non differentiable points and backprop will work fine. $\endgroup$ – Ray Apr 7 at 23:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.