How do intermediate layers of a trained neural network look like?

Question

Suppose I have a deep feed-forward neural network with sigmoid activation $\sigma$ already trained on a dataset $S$. Let's consider a training point $x_i \in S$. I want to analyze the entries of a hidden layer $h_{i,l}$, where

$$h_{i,l} = \sigma(W_l ( \sigma (W_{l-1} \sigma( \dots \sigma ( W_1 \cdot x_i))\dots). $$

My intuition would be that, since gradient descend has passed many times on the point $x_i$ updating the weights at every iteration, the entries of every hidden layer computed on $x_i$ would be either very close to zero or very close to one (thanks to the effect of the sigmoid activation).

Is this true? Is there a theoretical result in the literature which shows anything similar to this? Is there an empirical result which shows that?

Answer 1

tl;dr

Your intuition is correct.

Why is this a problem?

The saturating effect of the sigmoid activation function is well documented as it is the main culprit of a problem called vanishing gradients.

In short, the derivative of the sigmoid function is:

$$ \frac{dσ(z)}{dz} = σ(z) (1 - σ(z)) $$

The problem is that very often (especially if we initialize our weights with large values), the output of this neuron will either be $1$ or $0$. This causes the gradient to be $0$, which in turn means that the weights of that neuron can't be updated.

So let's keep the weights small (in absolute value)

Initially this seems like a good idea, if we keep the weights small we will avoid saturation.

However there is another problem with sigmoid functions: their gradient has a maximum value of $0.25$! This means that if the weight of a neuron is less than $4$, the error will diminish while flowing backward through the net. This becomes progressively worse as we add more layers to the network.

How have we solved this issue

Naturally researchers tried to find better weight initialization strategies. However this was hard, because as we saw, small weights are bad and large weights are bad.

One example is Mishkin et al. 2016 who propose a new initialization strategy, but fail to train a deep neural network with sigmoid activations.

Another workaround is to use a different learning rate for each layer (Xu et al. - Revise Saturated Activation Functions)

After a while the Machine Learning community realized that sigmoid functions were ill-suited for deep neural networks and adopted ReLU activations, which have fewer drawbacks and scale better. Nowadays, they have become the de-facto choice for deep learning.

Sources

This problem is known for several years and has been well documented (the earliest I could find was in 1994). This was mostly explored in Recurrent Neural Networks.

If you are interested in reading about this, I'd recommend this post, by Andrej Karpathy.

Some more formal sources on this topic:

• Hochreiter et al. - Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies
• Hochreiter et al. - Recurrent Neural Net Learning and Vanishing Gradient
• Pascanu et al. - On the difficulty of training recurrent neural networks
• Bengio et al. - Learning long-term dependencies with gradient descent is difficult