Xavier vs He initialization with tanh

Question

I'm a student and in the lecture, I learned that He initialization is better than Xavier if you use ReLU activation function.

In addition, I also learned that Xavier initialization is better than He initialization.

However, I once implemented a simple MLP with 3 hidden layers using tanh, for function approximation as below: 2 input neurons 8 hidden neurons 1 output neuron

In the result, He got better result than Xavier, which is not as I expected.

Their final loss was similar, but He converged faster.

Can anyone help me why this happens? uniform Xavier and norm Xavier did not showed big difference.

Answer 1

Xavier (or Glorot) initialization is designed to prevent both sigmoid and tanh activations to saturate, i.e., to yield values close to 0 or 1 (for sigmoid) or -1, +1 (for tanh) which causes the gradient during backprop to be zero or so, therefore halting learning. The same happens for the ReLU, when its output is zero. So, He init, again, tries to prevent that.

Now, for simple problems and shallow networks you can't really expect big differences, which should be larger for more complex problems (e.g., learn from images or high-dimensional data) and deeper networks, for which He init is also optimized for (think about ResNets that have up to hundreds of layers.)

In your case, it may have happened that the He init prevented the tanh to saturate (even better than Xavier). But to check that you should plot a per-layer (and per-neuron) histogram of the activations to see that values should not be at the extremes but rather around zero.