There is a popular story regarding the back-of-the-envelope calculation performed by a British physicist named G. I. Taylor. He used dimensional analysis to estimate the power released by the explosion of a nuclear bomb, simply by analyzing a picture that was released in a magazine at the time.

I believe many of you know some nice back-of-the-envelope calculations performed in machine learning (more specifically neural networks). Can you please share them?

  • $\begingroup$ Whasup with the pronoun change from singular first person to plural first person? $\endgroup$ – FreezePhoenix Apr 17 '18 at 15:09
  • $\begingroup$ Yeah, not sure why I did that. $\endgroup$ – Charles Apr 17 '18 at 15:36

I have one to share. This is no formula, but a general thing I have noticed.

The number of neurons + neurons should be proportionate, in some way, to the complexity of the classification.

Although this is fairly basic and widely known, it has helped me in many times to consider one thing: how many at a minimum does it need?

| improve this answer | |
  • $\begingroup$ Are you sure you understood his question? $\endgroup$ – DuttaA Apr 17 '18 at 15:17
  • 1
    $\begingroup$ @DuttaA now that you mention it, no. $\endgroup$ – FreezePhoenix Apr 17 '18 at 15:20
  • $\begingroup$ He is asking something like approximations by NN apparently without very useful information..like the speed of a car from a photo or the pitch of an animal/instrument from its pic..something like that I guess..though im not sure $\endgroup$ – DuttaA Apr 17 '18 at 15:23
  • $\begingroup$ @DuttaA I thought he was talking about semi-useful information about NN, AKA "tidbits". $\endgroup$ – FreezePhoenix Apr 17 '18 at 15:24
  • $\begingroup$ It looks like my question may be a bit vague. I’ll try to add more constraints this afternoon if I have time. $\endgroup$ – Charles Apr 17 '18 at 15:41

I think a nice back-of-the envelope calculation is the intuition for exploding/vanishing gradients in RNNs:

Simplifications: diagonalisable weights, no non-linearities, 1 layer: $h_t = W\cdot h_{t-1} + U\cdot x_t$

Let $L_t$ be the loss at timestep $t$ with the total loss $L = \sum_t L_t$

$$ \frac{\partial L_t}{\partial W} \sim= \sum_{k=1}^{t}\frac{\partial h_t}{\partial h_k}\times\alpha_{t, k} $$

Let's not care about terms regrouped in $\alpha_{t, k}$:

$$ \frac{\partial h_t}{\partial h_k} = \prod_{k<i\leq t} \frac{\partial h_i}{\partial h_{i-1}} = \prod_{k<i\leq t} W = \prod_{k<i\leq t} PDP^\top = PD^{t-k}P^{\top} $$

So you can easily see that if the eigen values of $W$ (in the diagonal matrix $D$) are larger than $1$, the gradient will explode with time, and if they are smaller than $1$, it will vanish.

More detailed derivations in On the difficulty of training recurrent neural networks

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.