Is the cube root function suitable as a n activation function?

Question

I am trying to design a neural network on Python.

Instead of the sigmoid function which has a limited range, I am thinking of using the cube root function which has the following graph:

Is this suitable?

Answer 1

There are a few traits that you want the activation function to have, and cube roots rate as OK-ish:

• Nonlinear – check.

• Continuously differentiable – no. There is a problem at $x=0$. Unlike other discontinuous functions like ReLU, although the gradient can be calculated near zero, it can be arbitrarily high as you approach $x=0$, because $\frac{d}{dx}x^{\frac{1}{3}} = \frac{1}{3x^{\frac{2}{3}}}$

• Range considerations – limited range functions are more stable, large/infinite range functions are more efficient. You may need to reduce learning rates compared to e.g. tanh.

• Monotonic – check.

• Monotonic derivative - no.

• Approximates identity near the origin – no, the approximation is bad near the origin.

If you look through the list of current, successful activation functions, you will see a few that also fail to provide one or more desirable traits, yet are still used routinely.

I would worry about the high gradients near $x=0$, but other than that I think the function could work ok. It may sometimes be unstable during learning, as small changes near zero will result in large changes to output. You might be able to work around the high gradients simply in practice, by clipping them. If the raw calculation returns value greater than $1$ (or less than $-1$) then treat the gradient as if it were $1$ (or $-1$) for the rest of backpropagation.

The only way of finding out if the function is competitive with other more standard activation functions is to try it on some standard data sets and make comparisons.