1
$\begingroup$

I am quite new to neural networks, and would like to save myself some of the learning curve by having some rules of thumb about hidden layer sizes.

I would also like to have a rule of thumb for the ratio between the size of the dataset and the size or depth of the network.

Currently, for example, I need to create an autoencoder for a dataset which is about 3000 images of a small size, say 25x25. I would assume the innermost layer would be of size ~10, but I don't really know why would it be either more or less, or what I should see in order to want to change it in either direction.
Same goes for any other layer.

Are there any thumb rules for network sizes? Are there any thumb rules for optimization of sizes? I find it hard to believe trial and error is all there is, but I found now course that teaches how to do that.

Improve this question
$\endgroup$
3
  • $\begingroup$ Similar questions have already been asked in the past. See e.g. this, this, this, this, this and this. If any of the answers to these questions answer your question, let me know. If not, please, clarify why in your post. $\endgroup$
    – nbro ♦
    Oct 5 '20 at 11:30
  • $\begingroup$ @nbro Well, it seems the answer is "trial and error". I was hoping something better would have come up by now. At least something to give a range of values. What is too high? What is too low? You can close this is you like. Thanks :) $\endgroup$
    – Gulzar
    Oct 5 '20 at 11:43
  • $\begingroup$ Maybe take a look at this post (I had provided an answer there that goes into the direction of your questions "What is too high? What is too low?". Maybe if you ask the question "Are there any bounds on the required capacity of an auto-encoder to perform some task X?". That would be a distinct enough question that you could ask. I suggest that you ask it separately, but you can also edit this one. $\endgroup$
    – nbro ♦
    Oct 5 '20 at 21:22