3
$\begingroup$

Raul Rojas' Neural Networks A Systematic Introduction, section 8.2.1 calculates the standard deviation of the output of a hidden neuron.

From: $$ \sigma^2 = \sum^n_{i=0}E[w_i^2]E[x_i^2] $$

When I try what I get is (with $E[x_i^2] = \frac{1}{3}$ and $w_i \in [-\alpha, \alpha]$):

$$ \sigma^2 = \sum^n_{i=0}E[w_i^2]E[x_i^2] = [n\frac{(\alpha-(-\alpha))^2}{12}][n\frac{1}{3}]=n^2\alpha^2\frac{1}{9} $$

$$ \sigma = \sqrt{n^2\alpha^2\frac{1}{9}}=n\alpha\frac{1}{3} $$

But Raul Rojas concludes:

$$ \sigma = \frac{1}{3}\sqrt{n}\alpha $$

Am I missing some implicance of the law of large numbers use for the input to the node?

Thank you for your time :)

$\endgroup$
1
$\begingroup$

If n different edges with associated weights $w_1, w_2, . . . , w_n$ point to this node, then after selecting weights with uniform probability from the interval $[−α, α]$, the expected total input to the node is $$\sum_iw_ix_i$$. By the law of large numbers we can also assume that the total input to the node has a Gaussian distribution.

Note that the input to a node is $$w_1x_1 + w_2x_2 + ... + w_nx_n $$

The variance of the total input to a node is:

$$\sigma^2 = \sum^n_{i=1}E[w_i^2]E[x_i^2] = n(E[w_1^2]E[x_1^2]) = n([\frac{(\alpha-(-\alpha))^2}{12}][\frac{1}{3}])=n\alpha^2\frac{1}{9}$$

since inputs and weights are uncorrelated.

I have taken out the summation and replaced it with $n$ times the variance of an input and a weight -- this can be done because both are independent and identically distributed in their own right and are independent of each other too. Variance of $w_1$ is the same as variance of $w_n$.

Am I missing some implicance of the law of large numbers use for the input to the node?

Yes.

Implication of Law of Large Numbers:

Note that the random variable $E(w_ix_i)$ in $$E(\sum^n_i(w_ix_i)^2)$$ is written as product of the random variables $E(w_i)$ and $E(x_i)$ $$\sum^n_{i=1}E[w_i^2]E[x_i^2]$$. This can only be written when the two random variables are independent. And the switching of expectation and summation is due to the linearity of expectation property. How do we get independence here? LLN comes to the rescue.

That comes from the fact that -- according to the law of large numbers -- the random variable (total input) follows a normal distribution which implies the two random variables i.e. the weights and the inputs also follow normal distribution -- and it is also given that the two random variables are uncorrelated and thus they are independent.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.