35
votes
How to select number of hidden layers and number of memory cells in an LSTM?
Your question is quite broad, but here are some tips.
Specifically for LSTMs, see this Reddit discussion Does the number of layers in an LSTM network affect its ability to remember long patterns?
...
- 1,177
19
votes
Accepted
How to find the optimal number of neurons per layer?
There is no direct way to find the optimal number of them: people empirically try and see (e.g., using cross-validation). The most common search techniques are random, manual, and grid searches.
...
- 2,556
16
votes
Accepted
How to choose an activation function for the hidden layers?
It seems to me that you already understand the shortcomings of ReLUs and sigmoids (like dead neurons in the case of plain ReLU).
You may want to look at ELU (exponential linear units) and SELU (self-...
- 943
14
votes
Accepted
Why should the number of neurons in a hidden layer be a power of 2?
I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster.
I ...
- 27k
11
votes
Accepted
How can I automate the choice of the architecture of a neural network for an arbitrary problem?
I think in this case, you'll probably want to use a genetic algorithm to generate a topology rather than working on your own. I personally like NEAT (NeuroEvolution of Augmenting Topologies).
The ...
- 226
8
votes
How to select number of hidden layers and number of memory cells in an LSTM?
The selection of the number of hidden layers and the number of memory cells in LSTM probably depends on the application domain and context where you want to apply this LSTM.
The optimal number of ...
- 816
7
votes
How to select number of hidden layers and number of memory cells in an LSTM?
In general, there are no guidelines on how to determine the number of layers or the number of memory cells in an LSTM.
The number of layers and cells required in an LSTM might depend on several ...
- 699
7
votes
How to find the optimal number of neurons per layer?
For a more intelligent approach than random or exhaustive searches, you could try a genetic algorithm such as NEAT http://nn.cs.utexas.edu/?neat. However, this has no guarantee to find a global optima,...
- 712
7
votes
Accepted
How do I decide the optimal number of layers for a neural network?
There is a technique called Pruning in neural networks, which is used just for this same purpose.
The pruning is done on the number of hidden layers. The process ...
- 1,371
7
votes
Accepted
Is this idea to calculate the required number of hidden neurons for a single hidden layer neural network correct?
I have an idea to find the optimal number of hidden neurons required in a neural network but I'm not sure how accurate it is.
It's a complete non-starter, and there is a no such calculation possible ...
- 27k
7
votes
Accepted
Do we need automatic hyper-parameter tuning when we have a large enough dataset?
Unfortunately, even with large amounts of training data, hyperparameter choices can strongly influence the performance of a trained model.
What you can usually drop when you have large amounts of ...
- 27k
6
votes
How to find the optimal number of neurons per layer?
Paper Szegedy C, Vanhoucke V, Ioffe S, et al. Rethinking the inception architecture for computer vision[J]. arXiv preprint arXiv:1512.00567, 2015. gives some general design principles:
Avoid ...
- 161
6
votes
How is neural architecture search performed?
You could say that NAS fits into the domain of Meta Learning or Meta Machine learning.
I've pulled the NAS papers from my notes, this is a collection of papers/lectures that I personally found very ...
5
votes
Accepted
How to shorten the development time of a neural network?
Your scenario is common.
The most straightforward approach is to subsample your data randomly. Unless your data or your model has strong bias, your performance to the smaller data set should be ...
- 1,411
5
votes
How can I automate the choice of the architecture of a neural network for an arbitrary problem?
The other answer mentions NEAT to generate network weights or topologies. The paper NeuroEvolution: The Importance of Transfer Function Evolution and Heterogeneous Networks, which also gives a short ...
- 7,186
5
votes
Accepted
In Q-learning, shouldn't the learning rate change dynamically during the learning phase?
Yes you can decay the learning rate in Q-learning, and yes this should result in more accurate Q-values in the long term for many environments.
However, this is something that is harder to manage ...
- 27k
5
votes
What is a "surrogate model"?
A surrogate model is a simplified model. It is a mapping $y_S=f_S(x)$ that approximates the original model $y=f(x)$, in a given domain, reasonably well. Source: Engineering Design via Surrogate ...
- 171
5
votes
Accepted
What is a "surrogate model"?
What is Bayesian optimization?
Introduction
Bayesian optimization (BO) is an optimization technique used to model an unknown (usually continuous) function $f: \mathbb{R}^d \rightarrow Y$, where ...
- 38.1k
5
votes
Accepted
Why do we need both the validation set and test set?
The difference between the validation and test set in my opinion should be explained in this way:
the validation set is meant to be used multiple times.
the test set is meant to be used only once.
...
- 4,948
5
votes
Accepted
What is the meaning of "exploration" in reinforcement and supervised learning?
In reinforcement learning, exploration has a specific meaning, which is in contrast with the meaning of exploitation, hence the so-called exploration-exploitation dilemma (or trade-off). You explore ...
- 38.1k
4
votes
How to select number of hidden layers and number of memory cells in an LSTM?
Have a look at the paper Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling (2014), where different LSTM architectures are compared. In the abstract, the ...
- 279
4
votes
Accepted
How can I avoid overfitting when doing parameter tuning?
Yes. Usually you would use cross validation to avoid overfitting during parameter tuning. If your dataset is large enough, and you don't try too many parameter combinations, this will work well, ...
- 9,097
4
votes
Accepted
Is there a reason to choose regular momentum over Nesterov momentum for neural networks?
The book Deep Learning by Goodfellow, Bengio, and Courville says (Sec 8.3.3, p 292 in my copy) states that
Unfortunately, in the stochastic gradient case, Nesterov momentum does not improve the ...
- 224
4
votes
How to determine the embedding size?
In most cases, seems that embedding dim is chosen empirically, by trial and error.
Older papers in NLP used 300 conventionally https://petuum.medium.com/embeddings-a-matrix-of-meaning-4de877c9aa27. ...
- 2,314
4
votes
Accepted
How do I design the network for Deep Q-Network?
What is the strategy to get to a better network?
There are a few different strategies that you can use to search for good hyperparameters in reinforcement learning RL, but you should be aware that ...
- 27k
3
votes
Accepted
How do I choose the size of the hidden state of a GRU?
Yes, your understanding of the hidden state is correct. But the size of the hidden state is a hyperparameter that needs to found by trial-and-error. There is no closed-form formula or solution which ...
- 562
3
votes
How should we choose the dimensions of the encoding layer in auto-encoders?
The number of dimensions is a hyperparameter of your model, and you should do a hyperparameter search, like with any other parameters. There's also a tradeoff between dimension and training speed, so ...
3
votes
Accepted
What is the pros and cons of increasing and decreasing the number of worker processes in A3C?
The correct number of child processes will depend on the hardware available to you.
Simplifying a bit, child processes can be in one of two states: waiting for memory or disk access, or running.
If ...
- 9,097
3
votes
How to choose an activation function for the hidden layers?
***Take my answer as a side note to that given by cantordust:
If one can verify that an activation function perform well in some cases, that good behavior often extrapolates to other problems.
Thus, ...
- 31
3
votes
Do genetic algorithms also evolve?
In principle, yes, you can also evolve the genetic algorithm (or, in general, evolutionary algorithm), i.e. you can evolve its operations (such as the mutation and cross-over) and hyper-parameters (...
- 38.1k
Only top scored, non community-wiki answers of a minimum length are eligible