A probability density function is a real-valued function that roughly gives the density of probability at a particular value of a random variable.

For example, the probability density function of a normal random variable is given by

$$f(x) = \dfrac{1}{2\sigma \sqrt{2\pi}} e^{-{\LARGE(}\dfrac{x-\mu}{\sigma}{\LARGE)}^2}$$

Uniform Kaiming He probability distribution function is used for initialization of weights in Convolutional neural networks in PyTorch and the distribution function was initially mentioned in the research paper titled Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification by Kaiming He et al. I think.

What is the analytical formula for the Kaiming He probability density function?


The activation function proposed by He et al. is not a new probability function of its own kind. It's an improvement over a previously proposed activation function now called Xavier or Glorot (even though it was named by the authors normalized activation in the original paper). The Xavier activation is also simply an activation function and not a new kind of probability distribution.

To answer the question, both functions simply sample values from either a uniform or normal distribution. What change is how the parameters that define the distributions are estimated.

Focusing on the Kaiming activation, since that's the question asked, the formulas used to determine the correct parameters of the uniform and normal distributions to sample values from are:

  • Uniform: $$\boldsymbol{\mathit{U}}(-\sqrt{6 / n_{j}}, +\sqrt{6 / n_{j}}) $$

  • Normal: $$\boldsymbol{\mathbf{\mathit{N}}}(0, \sqrt{2/n_{j}})$$

nj is the generic notation used in the original paper, in other sources like the pytorch documentation the notation used is fan_in or fan_out. Without explaining the details, fan_in means number of hidden units of the input layers for the weight matrix being initialized (or number of rows of the weight matrix to initialize), while fan_out means number of hidden units of the output layers for the weight matrix being initialized (or number of columns of the weight matrix to initialize).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.