In the research paper on Xavier initialization what is the purpose of putting $n_{in}$ and $n_{out}$ under 2 and added them it just says as a compromise but it is not exactly a harmonic mean or an average it seems random ? 
$\begingroup$
$\endgroup$
0
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
9
More of an extended comment but it felt worth giving as an answer.
- Equations (10) and (11) give you the estimates $\mathrm{Var}(W^i) = \frac{1}{n_i}$ and $\mathrm{Var}(W^i) = \frac{1}{n_{i+1}}$ respectively. Equation (12) is the harmonic mean of these estimates.
- As pointed out here, one simple way that Glorot and Bengio could have arrived at this "compromise" (without thinking about harmonic means) is to add together equations (10) and (11).
- The paper "Delving Deep into Rectifiers" by He-Zhang-Ren-Sun (PDF), which introduces Kaiming initialization, argues that it shouldn't actually matter if one uses $\frac{1}{n_i}$ or $\frac{1}{n_{i+1}}$ (or any number between them, like the harmonic mean). Here's the argument: suppose that we pick $\mathrm{Var}(W^i) = \frac{1}{n_i}$, so the signal variance is constant through the forward pass. Then over the backward pass, the variance of the gradients scales by a total of $$\prod_{i=1}^{L-1} \frac{n_{i+1}}{n_i} = \frac{n_L}{n_1},$$ where $L$ is the number of layers. For most typical neural network designs, this will be within an order of magnitude of 1; also, it won't change if the network gets deeper. So even if the gradients scale somewhat over the backward pass, the scaling isn't exponential and we don't expect vanishing/exploding gradient problems.
- As you hopefully realized, what's in the screenshot is just a calculation of variance, and we still have to choose a distribution. People typically use the uniform or normal distributions with mean 0 and the given variance. The 6 in the numerator of the formula for the "Xavier uniform" distribution is just there to give a distribution with the variance in (12).
-
1$\begingroup$ wow that was very helpful cleared up a lot of questions I had $\endgroup$– StefCommented Feb 23 at 21:06
-
$\begingroup$ Xavier initialization scheme with harmonic mean of in/out info flow rates is more conservative than He, even if the gradients variance scaling isn't exponential, in deep layers it may still very likely become large enough to cause sigmoid/tanh activations exploding or vanishing. He scheme is more appropriate for the more common Relu activations since Relu is more stable to become saturated comparatively. $\endgroup$– cinchCommented Feb 23 at 22:13
-
$\begingroup$ @cinch I think that the important difference between He and Xavier initializations is that the output of a ReLU has roughly half the variance as the output of a sigmoid/tanh, and so He et al. initialize the weights with roughly twice the variance. I would point you to p. 1030 of the He et al. paper (paragraph starting "We note that it is sufficient to use...") for the argument I gave in #3, which applies equally well to non-ReLU activations. $\endgroup$ Commented Feb 25 at 15:54
-
$\begingroup$ @PaulVanKoughnett can you please explain the part about how "this will be within an order of maginitude of 1 also it wont change as network get deeper" $\endgroup$– StefCommented Feb 27 at 20:25
-
1$\begingroup$ @Stef Yeah, that's right -- but more generally, if you pick numbers around the same size (e.g. the harmonic mean), then you have controlled variance change on the way forward and back, so you'll still be fine. (Also worth emphasizing that the whole argument only applies to the initial state of the network -- definitely possible for gradients to start vanishing later on in training.) $\endgroup$ Commented Feb 29 at 19:24
$\begingroup$
$\endgroup$
11
Note your case of numerator $2$ is chosen for normal Xavier initialization, there're other choice such as $6$ for uniform Xavier initialization. So the heuristic intuition behind the smaller number $2$ is related to applicable deep feedforward neural network architectures. When initializing weights for activation functions like sigmoid and tanh the normal Xavier initialization technique is suitable because these functions have derivatives that are relatively small compared to the linear activation functions such as relu. For inputs that are far from zero, the derivative of the sigmoid or and tanh function becomes very small or approaching zero to cause vanishing gradient or activation saturation problems, therefore the variance of initialized weights ought to be scaled smaller to keep activations within a reasonable range regardless of the layer sizes.
In summary the formula here is somewhat heuristic and there might be cases where other methods, or even different forms of the Xavier initialization, work well for specific architectures or datasets.
-
$\begingroup$ Could it also be that if one layer is much bigger than the other then that means the input or activation is being looked at in a bunch of different ways in the bigger layer so the variance of the weights decreases rapidly with the formula they used indicating that maybe you don’t need that much variance if you have a ton of neurons because the input or activation is being looked at in a bunch of different ways $\endgroup$– StefCommented Feb 19 at 15:33
-
$\begingroup$ See the Xavier paper: "For tanh networks, the proposed normalized initialization can be quite helpful, presumably because the layer-to-layer transformations maintain magnitudes of activations (flowing upward) and gradients (flowing backward)... we have used the same number of units per layer. However, we verified that we obtain the same gains when the layer size increases (or decreases) with layer number". Thus your intuition about the relative layer size is incorrect, the desirable thing for DL weight initialization is to maintain variance from above across layers such as Var(z)~=Var(X). $\endgroup$– cinchCommented Feb 19 at 22:43
-
$\begingroup$ In Xavier initialization, the key factor is the number of inputs and outputs in the layers and not so much the method of randomization. The goal is to maintain the variance in bounds that enable effective learning with various activation functions. $\endgroup$– cinchCommented Feb 19 at 22:47
-
1$\begingroup$ Now you're correct. The ith layerwise variance is derived by equation (5) in the same paper, so to achieve above variance stability goal equation (10) which is same as your expression must hold for each layer. Hope this clarifies and helpful. $\endgroup$– cinchCommented Feb 19 at 23:37
-
1$\begingroup$ Obviouslya your specifically concerned formula is not exactly harmonic mean of nin and nout but resemble harmonic mean, as explained in my answer it's mainly derived empirically to strike a balance between avoiding vanishing gradients (if var(z) is too small) and preventing exploding gradients (if var(z) is too large). Comments are already too long. Please accept and close the question if you're satisfied. $\endgroup$– cinchCommented Feb 20 at 0:07