Conventionally, (although there are plenty of better options) it is being said that as the choice of activation function for hidden layers, tanh should be prefered over sigmoid because it has a zero mean but what if the data at hand is 0.5 mean and we are not willing to zero mean the data? Would tanh still be the go-to option?
$\begingroup$
$\endgroup$
1
-
1$\begingroup$ i dont think it's important, sigmoid is just as fine as tanh $\endgroup$– Dan D.May 9, 2022 at 3:49
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
The first part of this answer is regarding your concrete question and the second part summarizes things on activation functions in general because I believe that there are more important factors than what exactly the output spaces of the functions are, for the following reasons:
Should activation functions be zero-meaned:
I would say that in general, it is neither beneficial nor important for an activation function to be zero-mean.
but what if the data at hand is 0.5 mean and we are not willing to zero mean the data? Would
tanhstill be the go-to option?
IMO this intuition is a bit off because the activation functions are not applied to the data space directly, but to the latent spaces that are learnable:
The matrix multiplication of the first layer will be applied prior to the activation function and it will change the topology of the data, hence also attributes like mean and variance. So your (let's say 2-dimensional) input data can have a mean of [-1, -1], but if the matrix of the first layer learns something like a 180° rotation, the mean of the input to the activation function is now [1, 1]. Therefore, I would say that it is not so important for the choice of activation function what the mean of your data is.
However, in certain cases, it may have some useful properties when the latent representations of your data are zero-meaned, but I would expect this to occur in shallow models rather than deep ones and I would say that this is something to find out empirically for the individual problem.
Disclaimer I don't think this question can be answered definitively, however, this is my understanding of the matter, happy to get comments and remarks on this.
Some facts regarding sigmoid $(\sigma)$ and tanh
Properties of the sigmoid $(\sigma)$
(-) The maximum of the derivative of the $\sigma$ is rather small ($\max \sigma'(x) = 0.25$) which can slow down training and prevent deep models from training at all due to the vanishing gradient problem. (s. derivative of
sigmoidand derivative oftanh)(-) The sigmoid saturates towards $-\infty$ and $+\infty$, thus the gradient becomes small when the pre-activation values are very positive or very negative values, this is also a cause for vanishing gradient and prevents the training of deep models.
(+) A very useful property is that $\sigma(x) \in (0, 1)$. This is the reason why it is commonly used as an output activation for binary classification models. For the same reason, it is the function that is used in gates of recurrent models (s. GRU-cell). So the sigmoid in this case can act as a continuous switch to determine the influence that, e.g. newly input features have on the hidden state of the model.
Properties of the tanh
(-) The
tanhsaturates towards $-\infty$ and $+\infty$ which poses similar drawbacks as for the sigmoid.(o) The
tanhcan provide a larger gradient than the sigmoid ($\max \tanh'(x) = 1.0$), although there are other functions like the ReLU function that will output a large gradient more consistently.(+) There probably exist some regression problems for which
tanhis a good match as an output activation.(+) The $\tanh(x) \in (-1, 1)$ this is probably one reason why it is commonly used as the activation function inside RNNs (vanilla, GRU, LSTM). This ensures that the hidden state will always be in this range and doesn't explode after several processing steps.
What about the ReLU? The ReLU function is - to my knowledge - the most commonly used activation function for hidden layers (excluding RNNs). It consistently provides a gradient of one which leads to faster learning, especially for deep models. There are problems like dead neurons that occur when individual neurons produce only negative values (pre-activation) in which case the gradient and the output of that neuron will be zero til eternity. However, there are methods to reduce this problem, like controlling the bias, choosing a smaller learning rate, or switching to the Leaky-ReLu that can recover from 'dead' neurons.
