4
$\begingroup$

Suppose I have a deep feed-forward neural network with sigmoid activation $\sigma$ already trained on a dataset $S$. Let's consider a training point $x_i \in S$. I want to analyze the entries of a hidden layer $h_{i,l}$, where

$$h_{i,l} = \sigma(W_l ( \sigma (W_{l-1} \sigma( \dots \sigma ( W_1 \cdot x_i))\dots). $$

My intuition would be that, since gradient descend has passed many times on the point $x_i$ updating the weights at every iteration, the entries of every hidden layer computed on $x_i$ would be either very close to zero or very close to one (thanks to the effect of the sigmoid activation).

Is this true? Is there a theoretical result in the literature which shows anything similar to this? Is there an empirical result which shows that?

$\endgroup$
  • $\begingroup$ Interesting question, I think Tensorflow and similar frameworks have mechanisms to take a look under the hood, so you might find answers with a relatively simple experiment. I'm not sure whether your intuition is right, but that's just my intuition :-) $\endgroup$ – Hans-Martin Mosner Mar 28 at 10:31
  • $\begingroup$ Weights in trained image classifier network have Gaussian distribution for each layer, with mean/var not close to 1, but that is my own anecdotal experience. I think it was supported by 1 paper, but I don't recall its title. PS even if initial weight were not Gaussian $\endgroup$ – mirror2image Jun 25 at 5:03
1
$\begingroup$

tl;dr

Your intuition is correct.

Why is this a problem?

The saturating effect of the sigmoid activation function is well documented as it is the main culprit of a problem called vanishing gradients.

In short, the derivative of the sigmoid function is:

$$ \frac{dσ(z)}{dz} = σ(z) (1 - σ(z)) $$

The problem is that very often (especially if we initialize our weights with large values), the output of this neuron will either be $1$ or $0$. This causes the gradient to be $0$, which in turn means that the weights of that neuron can't be updated.

So let's keep the weights small (in absolute value)

Initially this seems like a good idea, if we keep the weights small we will avoid saturation.

However there is another problem with sigmoid functions: their gradient has a maximum value of $0.25$! This means that if the weight of a neuron is less than $4$, the error will diminish while flowing backward through the net. This becomes progressively worse as we add more layers to the network.

How have we solved this issue

Naturally researchers tried to find better weight initialization strategies. However this was hard, because as we saw, small weights are bad and large weights are bad.

One example is Mishkin et al. 2016 who propose a new initialization strategy, but fail to train a deep neural network with sigmoid activations.

Another workaround is to use a different learning rate for each layer (Xu et al. - Revise Saturated Activation Functions)

After a while the Machine Learning community realized that sigmoid functions were ill-suited for deep neural networks and adopted ReLU activations, which have fewer drawbacks and scale better. Nowadays, they have become the de-facto choice for deep learning.

Sources

This problem is known for several years and has been well documented (the earliest I could find was in 1994). This was mostly explored in Recurrent Neural Networks.

If you are interested in reading about this, I'd recommend this post, by Andrej Karpathy.

Some more formal sources on this topic:

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.