Skip to main content
37 votes

How to select number of hidden layers and number of memory cells in an LSTM?

Your question is quite broad, but here are some tips. Specifically for LSTMs, see this Reddit discussion Does the number of layers in an LSTM network affect its ability to remember long patterns? ...
Thomas Wagenaar's user avatar
19 votes
Accepted

How to find the optimal number of neurons per layer?

There is no direct way to find the optimal number of them: people empirically try and see (e.g., using cross-validation). The most common search techniques are random, manual, and grid searches. ...
Franck Dernoncourt's user avatar
16 votes
Accepted

How to choose an activation function for the hidden layers?

It seems to me that you already understand the shortcomings of ReLUs and sigmoids (like dead neurons in the case of plain ReLU). You may want to look at ELU (exponential linear units) and SELU (self-...
cantordust's user avatar
14 votes
Accepted

Why should the number of neurons in a hidden layer be a power of 2?

I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster. I ...
Neil Slater's user avatar
  • 32.7k
10 votes

How large should the replay buffer be?

In order for the algorithm to have stable behavior, the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything. The larger the ...
nbro's user avatar
  • 40.9k
10 votes

How large should the replay buffer be?

You need to read this 2020 paper by Deepmind: "Revisiting Fundamentals of Experience Replay" They explicitly test the size of the experience replay, the replay-ratio of each experience and ...
Kari's user avatar
  • 270
9 votes

How to select number of hidden layers and number of memory cells in an LSTM?

The selection of the number of hidden layers and the number of memory cells in LSTM probably depends on the application domain and context where you want to apply this LSTM. The optimal number of ...
Maheshwar Ligade's user avatar
7 votes

How to select number of hidden layers and number of memory cells in an LSTM?

In general, there are no guidelines on how to determine the number of layers or the number of memory cells in an LSTM. The number of layers and cells required in an LSTM might depend on several ...
naive's user avatar
  • 709
7 votes

How to find the optimal number of neurons per layer?

For a more intelligent approach than random or exhaustive searches, you could try a genetic algorithm such as NEAT http://nn.cs.utexas.edu/?neat. However, this has no guarantee to find a global optima,...
Tim Atkinson's user avatar
7 votes
Accepted

Is this idea to calculate the required number of hidden neurons for a single hidden layer neural network correct?

I have an idea to find the optimal number of hidden neurons required in a neural network but I'm not sure how accurate it is. It's a complete non-starter, and there is a no such calculation possible ...
Neil Slater's user avatar
  • 32.7k
6 votes

How to find the optimal number of neurons per layer?

Paper Szegedy C, Vanhoucke V, Ioffe S, et al. Rethinking the inception architecture for computer vision[J]. arXiv preprint arXiv:1512.00567, 2015. gives some general design principles: Avoid ...
Dale's user avatar
  • 161
6 votes

What causes a model to require a low learning rate?

Gradient Descent is a method to find the optimum parameter of the hypothesis or minimize the cost function. where alpha is learning rate If the learning rate is high then it can overshoot the ...
Posi2's user avatar
  • 368
6 votes
Accepted

Should I be decaying the learning rate and the exploration rate in the same manner?

First of all, I'd say that there is a reason to give Learning Rate (LR) and Exploration Rate (ER) the same decay: they play at the same scale (the number of successive batches you'll train your model ...
16Aghnar's user avatar
  • 601
6 votes

How to determine the embedding size?

In most cases, seems that embedding dim is chosen empirically, by trial and error. Older papers in NLP used 300 conventionally https://petuum.medium.com/embeddings-a-matrix-of-meaning-4de877c9aa27. ...
spiridon_the_sun_rotator's user avatar
6 votes

How to determine the embedding size?

There is a rule of thumb that says min(50, num_categories/2). But this tops out at 100 categories, what to do after that? I propose this: When num_categories <= 1000: ...
aboveandbeyondis's user avatar
5 votes
Accepted

Why is the number of output channels 16 in the hidden layer of this CNN?

I understand your question as: "How did the author select the number of neurons in their hidden layer?" The number of neurons in the hidden layer is how you can control the complexity of the function ...
JahKnows's user avatar
  • 470
5 votes

What are the best hyper-parameters to tune in reinforcement learning?

You should read this study https://arxiv.org/abs/2006.05990 which does some empirical study on this question, specifically for on-policy, continuous action space DRL. It suggests that discount factor ...
Taw's user avatar
  • 1,281
4 votes

How to select number of hidden layers and number of memory cells in an LSTM?

Have a look at the paper Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling (2014), where different LSTM architectures are compared. In the abstract, the ...
Dieshe's user avatar
  • 289
4 votes
Accepted

What is relation between gradient descent and regularization in deep learning?

Usually, when talking about regularization for neural networks there are 3 main types: L1, L2 and dropout. All affect the gradient descent procedure. L1 and L2 regularization is implemented in the ...
user10283726's user avatar
4 votes
Accepted

For episodic tasks with an absorbing state, why can't we both have $\gamma=1$ and $T= \infty$ in the definition of the return?

$T = \infty$ and $\gamma = 1$ cannot be both true at the same time because the return defined in equation 3.11 is supposed to be a unified definition of the return for both continuing and episodic ...
nbro's user avatar
  • 40.9k
4 votes

What are the best hyper-parameters to tune in reinforcement learning?

Personally, I would choose the following two as the most important: epsilon: When using an epsilon-greedy policy, epsilon determines how often the agent should explore and how often it should exploit....
devidduma's user avatar
  • 552
4 votes

When can I call an entity a hyperparameter?

In older machine learning literature the given definition of hyperparameters was explicitly the same used in Bayesian statistics, i.e. a hyperparameter is a parameter of a prior distribution For ...
Edoardo Guerriero's user avatar
4 votes

Is it true that batch size of form $2^k$ gives better results?

The choice of the batch size to be a power of 2 is not due the quality of predictions . The larger the batch_size is - the better is the estimate of the gradient, but a noise can be beneficial to ...
spiridon_the_sun_rotator's user avatar
3 votes
Accepted

Is a calculus or ML approach to varying learning rate as a function of loss and epoch been investigated?

Has this been done? Difficult to prove a negative, but I suspect although plenty of research has been done into finding ideal learning rate values (the need for learning rate at all is an annoyance), ...
Neil Slater's user avatar
  • 32.7k
3 votes
Accepted

What is the pros and cons of increasing and decreasing the number of worker processes in A3C?

The correct number of child processes will depend on the hardware available to you. Simplifying a bit, child processes can be in one of two states: waiting for memory or disk access, or running. If ...
John Doucette's user avatar
3 votes

How to choose an activation function for the hidden layers?

***Take my answer as a side note to that given by cantordust: If one can verify that an activation function perform well in some cases, that good behavior often extrapolates to other problems. Thus, ...
pbp's user avatar
  • 31
3 votes

How should we choose the dimensions of the encoding layer in auto-encoders?

The number of dimensions is a hyperparameter of your model, and you should do a hyperparameter search, like with any other parameters. There's also a tradeoff between dimension and training speed, so ...
Konstantin Solomatov's user avatar
3 votes
Accepted

How do I design a neural network that breaks a 5-letter word into its corresponding syllables?

I would highly recommend modeling things differently with regard to how letters are presented to the model. While the problem is more natural, perhaps, for a Convolutional or Recurrent Neural Network, ...
Gilad Tsur's user avatar
3 votes
Accepted

How are training hyperparameters determined for large models?

In general, it is definitely very computationally expensive, so an exhaustive search is not performed in practice. However, there are some recent approaches for determining whether the architecture is ...
spiridon_the_sun_rotator's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible