I watched in the video lecture of cs224: Stanford CS224N: NLP with Deep Learning | Winter 2019 | Lecture 2 – Word Vectors and Word Senses.

They take the sample size of the window to be $2^5 = 32$ or $2^6 = 64$. Why is the sample size of stochastic gradient descent a power of 2? Why not we can take 42 or 53 as the sample window size?

Btw, how to identify best minimum window sample size?

  • 3
    $\begingroup$ I'm not sure of a specific reason, but it could just be that using powers of 2 is computationally and memory efficient, as information in a computer is stored as binary, so in powers of 2. If you use a number such as 53, there's a bunch of under-utilised bytes. It's generally good practice to stick to powers of 2 for most computational things if you have the choice. $\endgroup$
    – Recessive
    Mar 3 '20 at 6:28

You can take any size you want since most memories are now larger than that, if you were to map the indices to a register, 64 values will take 6 bits whereas we have registers upto 64 bits, so it wouldn't matter much even if you choose a different sample size.

As far as resolution is concerned, layers like maxpool which reduce the size by half (2x2 kernel is generally used), the resolutions of outputs of layers are multiplied/divided by powers of 2, so keeping that in mind, resolutions as power of 2 makes sense, but then again, we have padding and effects of convolution on borders.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.