When we take a look at the literature there are so many opinions. I was wondering what are some generally good practices to design an architecture, like how much depth would you prefer and how much width would you prefer. Should the number of training data influence your decisions of designing the architecture. What should the number of parameters be ? etc.

  • $\begingroup$ Can you please provide 2-3 examples of "literature" that are contradicting each other by citing them and quoting the relevant parts? $\endgroup$
    – nbro
    Sep 28 at 12:08
  • $\begingroup$ @nbro stats.stackexchange.com/a/223637/272525 and arxiv.org/abs/1605.07146 $\endgroup$ Sep 28 at 12:11
  • $\begingroup$ Can you please edit your post to quote (provide 2 sentences) that are contradicting themselves? $\endgroup$
    – nbro
    Sep 28 at 12:16

I am not sure if there really are contradicting opinions on this matter. CNNs, RNNs, LSTMs all have specific types of data they are good at predicting. Depth and width, or in general the size of the neural network mostly depends on the size of your dataset. You don't want to build a too large network that will overfit the available data, which can usually be understood to be the case after running a few epochs but in general, number of trainable parameters should be smaller than to total number of independent data points you have to avoid memorizing the data definitely.

Width and depth of the network depends on the size of the data available (as mentioned before), but can simply be thought just as another hyper-parameter that is to be decided by training itself. It can be defined as a part of cross validation process of building a neural network.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.