37
votes
How to select number of hidden layers and number of memory cells in an LSTM?
Your question is quite broad, but here are some tips.
Specifically for LSTMs, see this Reddit discussion Does the number of layers in an LSTM network affect its ability to remember long patterns?
...
- 1,217
19
votes
Accepted
How to find the optimal number of neurons per layer?
There is no direct way to find the optimal number of them: people empirically try and see (e.g., using cross-validation). The most common search techniques are random, manual, and grid searches.
...
- 3,112
16
votes
Accepted
How to choose an activation function for the hidden layers?
It seems to me that you already understand the shortcomings of ReLUs and sigmoids (like dead neurons in the case of plain ReLU).
You may want to look at ELU (exponential linear units) and SELU (self-...
- 953
14
votes
Accepted
Why should the number of neurons in a hidden layer be a power of 2?
I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster.
I ...
- 32.9k
11
votes
How large should the replay buffer be?
You need to read this 2020 paper by Deepmind:
"Revisiting Fundamentals of Experience Replay"
They explicitly test the size of the experience replay, the replay-ratio of each experience and ...
- 280
10
votes
How large should the replay buffer be?
In order for the algorithm to have stable behavior, the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything.
The larger the ...
- 41.1k
9
votes
How to select number of hidden layers and number of memory cells in an LSTM?
The selection of the number of hidden layers and the number of memory cells in LSTM probably depends on the application domain and context where you want to apply this LSTM.
The optimal number of ...
- 826
7
votes
How to select number of hidden layers and number of memory cells in an LSTM?
In general, there are no guidelines on how to determine the number of layers or the number of memory cells in an LSTM.
The number of layers and cells required in an LSTM might depend on several ...
- 709
7
votes
How to find the optimal number of neurons per layer?
For a more intelligent approach than random or exhaustive searches, you could try a genetic algorithm such as NEAT http://nn.cs.utexas.edu/?neat. However, this has no guarantee to find a global optima,...
- 712
7
votes
Accepted
Is this idea to calculate the required number of hidden neurons for a single hidden layer neural network correct?
I have an idea to find the optimal number of hidden neurons required in a neural network but I'm not sure how accurate it is.
It's a complete non-starter, and there is a no such calculation possible ...
- 32.9k
7
votes
How to determine the embedding size?
In most cases, seems that embedding dim is chosen empirically, by trial and error.
Older papers in NLP used 300 conventionally https://petuum.medium.com/embeddings-a-matrix-of-meaning-4de877c9aa27. ...
- 2,724
7
votes
How to determine the embedding size?
There is a rule of thumb that says min(50, num_categories/2). But this tops out at 100 categories, what to do after that? I propose this:
When num_categories <= 1000:
...
6
votes
How to find the optimal number of neurons per layer?
Paper Szegedy C, Vanhoucke V, Ioffe S, et al. Rethinking the inception architecture for computer vision[J]. arXiv preprint arXiv:1512.00567, 2015. gives some general design principles:
Avoid ...
- 161
6
votes
What causes a model to require a low learning rate?
Gradient Descent is a method to find the optimum parameter of the hypothesis or minimize the cost function.
where alpha is learning rate
If the learning rate is high then it can overshoot the ...
- 368
6
votes
Accepted
Should I be decaying the learning rate and the exploration rate in the same manner?
First of all, I'd say that there is a reason to give Learning Rate (LR) and Exploration Rate (ER) the same decay: they play at the same scale (the number of successive batches you'll train your model ...
- 601
6
votes
What are the best hyper-parameters to tune in reinforcement learning?
You should read this study https://arxiv.org/abs/2006.05990 which does some empirical study on this question, specifically for on-policy, continuous action space DRL.
It suggests that discount factor ...
- 1,301
5
votes
Accepted
Why is the number of output channels 16 in the hidden layer of this CNN?
I understand your question as: "How did the author select the number of neurons in their hidden layer?"
The number of neurons in the hidden layer is how you can control the complexity of the function ...
- 470
5
votes
What are the best hyper-parameters to tune in reinforcement learning?
Personally, I would choose the following two as the most important:
epsilon: When using an epsilon-greedy policy, epsilon determines how often the agent should explore and how often it should exploit....
- 562
4
votes
How to select number of hidden layers and number of memory cells in an LSTM?
Have a look at the paper Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling (2014), where different LSTM architectures are compared. In the abstract, the ...
- 289
4
votes
Accepted
What is relation between gradient descent and regularization in deep learning?
Usually, when talking about regularization for neural networks there are 3 main types:
L1, L2 and dropout. All affect the gradient descent procedure.
L1 and L2 regularization is implemented in the ...
- 68
4
votes
Accepted
For episodic tasks with an absorbing state, why can't we both have $\gamma=1$ and $T= \infty$ in the definition of the return?
$T = \infty$ and $\gamma = 1$ cannot be both true at the same time because the return defined in equation 3.11 is supposed to be a unified definition of the return for both continuing and episodic ...
- 41.1k
4
votes
When can I call an entity a hyperparameter?
In older machine learning literature the given definition of hyperparameters was explicitly the same used in Bayesian statistics, i.e.
a hyperparameter is a parameter of a prior distribution
For ...
- 5,378
4
votes
Is it true that batch size of form $2^k$ gives better results?
The choice of the batch size to be a power of 2 is not due the quality of predictions .
The larger the batch_size is - the better is the estimate of the gradient, but a noise can be beneficial to ...
- 2,724
3
votes
Accepted
Is a calculus or ML approach to varying learning rate as a function of loss and epoch been investigated?
Has this been done?
Difficult to prove a negative, but I suspect although plenty of research has been done into finding ideal learning rate values (the need for learning rate at all is an annoyance), ...
- 32.9k
3
votes
Accepted
What is the pros and cons of increasing and decreasing the number of worker processes in A3C?
The correct number of child processes will depend on the hardware available to you.
Simplifying a bit, child processes can be in one of two states: waiting for memory or disk access, or running.
If ...
- 9,342
3
votes
How to choose an activation function for the hidden layers?
***Take my answer as a side note to that given by cantordust:
If one can verify that an activation function perform well in some cases, that good behavior often extrapolates to other problems.
Thus, ...
- 31
3
votes
How should we choose the dimensions of the encoding layer in auto-encoders?
The number of dimensions is a hyperparameter of your model, and you should do a hyperparameter search, like with any other parameters. There's also a tradeoff between dimension and training speed, so ...
3
votes
Accepted
How do I design a neural network that breaks a 5-letter word into its corresponding syllables?
I would highly recommend modeling things differently with regard to how letters are presented to the model. While the problem is more natural, perhaps, for a Convolutional or Recurrent Neural Network, ...
- 46
3
votes
Accepted
How are training hyperparameters determined for large models?
In general, it is definitely very computationally expensive, so an exhaustive search is not performed in practice. However, there are some recent approaches for determining whether the architecture is ...
- 2,724
Only top scored, non community-wiki answers of a minimum length are eligible