Why don't we use a trigonometric function, such as $\tan(x)$, where $x$ is an element of the interval $[0,pi/2)$, instead of the sigmoid function for the output neurons (in the case of classification)?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ Are you only interested in classification or also regression? $\endgroup$– nbroCommented Jun 23, 2020 at 11:03
-
$\begingroup$ In classification. I'm curious as to why we fit the data using a sigmoid function. $\endgroup$– AC18Commented Jun 23, 2020 at 11:04
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Although it's true that if you use certain trigonometric functions, such as the tangent, you could have numerical problems (as suggested in this answer), that's not the only reason for not using trigonometric functions.
Trigonometric functions are periodic. In general, we may not want to convert a non-periodic function to a periodic one. To be more concrete, let's suppose we use the sine function as the activation function of the output neurons of a neural network. Assuming only one input, if the input to any of those output neurons is $360k$, for any integer $k$, the result will always be $0$, but that may not be desirable.
$\begingroup$
$\endgroup$
The main reason why the sigmoid function is used is because it 'does not blow up' since it is between 0 and 1 always. As for the relu it is used because it is computationally cheap and even resolves the problem of vanishing gradient(and hence used more often than sigmoid).
So a function like tan(x) will blow up for certain values of x. This can cause a problem of exploding gradients. So, I believe for this reason tan(x) cannot be a good non-linearity to be used.
As for any other function, it is more because of the results that we have gotten over the years and sigmoid and relu have been promising.
