5
$\begingroup$

It is suggested that the number of hidden units in a layer should be in powers of 2 because it helps converge faster. Is it a fact and if it is, how this helps the NN learn faster. Does it have to do something with how the memory is laid down?

$\endgroup$
  • 1
    $\begingroup$ I have completed that specialisation, and do not recall any suggestion that it "helps converge faster". Could you give a more precise reference (e.g. link to lecture or notes where this is said)? I'm pretty sure something else was said about using powers of 2 - along the lines of not bothering with small changes when searching hyperparameters. $\endgroup$ – Neil Slater Feb 22 '18 at 17:21
  • $\begingroup$ @NeilSlater I couldn't find the video, I'll fix the question. maybe I've heard it someplace else or read it somewhere but does it make sense 'cos I couldn't think of any reason why $\endgroup$ – dsfx3d Feb 22 '18 at 17:53
  • $\begingroup$ The idea is, that the hidden neurons are trained for a certain task. For example, the neuron 0-1 are doing a recognition of feature A, while the neurons 2-7 are trained on feature B. If the number of neurons is equal to “x^2” it is easier to group them. For example the overall number of 16 neurons can grouped into: 16=2*4+1*8 neurons. $\endgroup$ – Manuel Rodriguez Feb 23 '18 at 5:27
6
$\begingroup$

It is suggested that the number of hidden units in a layer should be in powers of 2 because it helps converge faster.

I would quite like to see a reference to this suggestion, in case it has been misunderstood.

As far as I know, there is no such effect in normal neural networks. In convolutional neural networks it might potentially be true in a minor way because some FFT approaches work better with 2^n items.

Is it a fact and if it is, how this helps the NN learn faster.

I would say that this is not a general fact. Instead, it seems like misunderstood advice to search some hyperparameters such as number of neurons in each layer, by increasing or decreasing by a factor of 2. Doing this and trying layer sizes of 32, 64, 128 etc should increase the speed of finding a good layer size compared to trying sizes 32, 33, 34 etc.

The main reason to pick powers of 2 is tradition in computer science. Provided there is no driver to pick other specific numbers, may as well pick a power of 2 . . . but equally you will see researchers picking multiples of 10, 100 or 1000 as "round numbers", for a similar reason.

One related factor: If a researcher presents a result for some new technique where the hidden layer sizes were tuned to e.g. 531, 779, 282 etc, then someone reviewing the work would ask the obvious question "Why?" - such numbers might imply the new technique is not generic or requires large amounts of hyperparameter tuning, neither of which would be seen as positive traits. Much better to be seen using an obvious "simple" number . . .

$\endgroup$
  • $\begingroup$ I think this answer is quite convincing. Thanks $\endgroup$ – dsfx3d Feb 23 '18 at 18:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.