Raul Rojas' Neural Networks A Systematic Introduction, section 8.2.1 calculates the variance of the output of a hidden neuron.

Raul Rojas says that "for binary vectors we have $E[x_i^2] = \frac{1}{3}$" where $x_i$ is the input value transported through each edge to a node.

I don't quite get how he reaches this result.

Thank you for your time :)

  • $\begingroup$ The integral from 0 to 1 of x^2dx is 1/3, but isn't this related to a continuous random variable, instead of a binary one? $\endgroup$ – Jaume Oliver Lafont Apr 24 '19 at 4:47

Some lines above the author says

By the law of large numbers we can also assume that the total input to the node has a Gaussian distribution

hence we can assume $X \sim \mathcal{N}(0,1)$ with the $X$ domain being continuous

Then he says the input vector is assumed to be binary which changes the domain from continuous to discrete so we can discretize it assuming $-1 \le X \le 1$ is mapped into zero and $X< -1 $ and $X > 1$ are mapped to 1

Finally according to the 68-95-99.7 Rule we can compute

$$ E(X^2) = P(X=1) \cdot 1^2 + P(X=0) \cdot 0^2 = 0.32 $$

Finally probably the author rounds this up to $\frac{1}{3} \simeq 0.33$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.