30 votes

How to select number of hidden layers and number of memory cells in an LSTM?

Your question is quite broad, but here are some tips. Specifically for LSTMs, see this Reddit discussion Does the number of layers in an LSTM network affect its ability to remember long patterns? ...
user avatar
19 votes
Accepted

How to find the optimal number of neurons per layer?

There is no direct way to find the optimal number of them: people empirically try and see (e.g., using cross-validation). The most common search techniques are random, manual, and grid searches. ...
user avatar
15 votes
Accepted

How to choose an activation function for the hidden layers?

It seems to me that you already understand the shortcomings of ReLUs and sigmoids (like dead neurons in the case of plain ReLU). You may want to look at ELU (exponential linear units) and SELU (self-...
user avatar
14 votes
Accepted

Why should the number of neurons in a hidden layer be a power of 2?

I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster. I ...
user avatar
11 votes
Accepted

How can I automate the choice of the architecture of a neural network for an arbitrary problem?

I think in this case, you'll probably want to use a genetic algorithm to generate a topology rather than working on your own. I personally like NEAT (NeuroEvolution of Augmenting Topologies). The ...
user avatar
  • 226
8 votes

How to select number of hidden layers and number of memory cells in an LSTM?

The selection of the number of hidden layers and the number of memory cells in LSTM probably depends on the application domain and context where you want to apply this LSTM. The optimal number of ...
user avatar
7 votes

How to select number of hidden layers and number of memory cells in an LSTM?

In general, there are no guidelines on how to determine the number of layers or the number of memory cells in an LSTM. The number of layers and cells required in an LSTM might depend on several ...
user avatar
  • 699
7 votes

How to find the optimal number of neurons per layer?

For a more intelligent approach than random or exhaustive searches, you could try a genetic algorithm such as NEAT http://nn.cs.utexas.edu/?neat. However, this has no guarantee to find a global optima,...
user avatar
7 votes
Accepted

How do I decide the optimal number of layers for a neural network?

There is a technique called Pruning in neural networks, which is used just for this same purpose. The pruning is done on the number of hidden layers. The process ...
user avatar
  • 1,351
7 votes
Accepted

Is this idea to calculate the required number of hidden neurons for a single hidden layer neural network correct?

I have an idea to find the optimal number of hidden neurons required in a neural network but I'm not sure how accurate it is. It's a complete non-starter, and there is a no such calculation possible ...
user avatar
6 votes

How to find the optimal number of neurons per layer?

Paper Szegedy C, Vanhoucke V, Ioffe S, et al. Rethinking the inception architecture for computer vision[J]. arXiv preprint arXiv:1512.00567, 2015. gives some general design principles: Avoid ...
user avatar
  • 161
6 votes

How is neural architecture search performed?

You could say that NAS fits into the domain of Meta Learning or Meta Machine learning. I've pulled the NAS papers from my notes, this is a collection of papers/lectures that I personally found very ...
user avatar
6 votes
Accepted

Do we need automatic hyper-parameter tuning when we have a large enough dataset?

Unfortunately, even with large amounts of training data, hyperparameter choices can strongly influence the performance of a trained model. What you can usually drop when you have large amounts of ...
user avatar
5 votes
Accepted

How to shorten the development time of a neural network?

Your scenario is common. The most straightforward approach is to subsample your data randomly. Unless your data or your model has strong bias, your performance to the smaller data set should be ...
user avatar
  • 1,390
5 votes

How can I automate the choice of the architecture of a neural network for an arbitrary problem?

The other answer mentions NEAT to generate network weights or topologies. The paper NeuroEvolution: The Importance of Transfer Function Evolution and Heterogeneous Networks, which also gives a short ...
user avatar
5 votes

What is a "surrogate model"?

A surrogate model is a simplified model. It is a mapping $y_S=f_S(x)$ that approximates the original model $y=f(x)$, in a given domain, reasonably well. Source: Engineering Design via Surrogate ...
user avatar
5 votes
Accepted

What is a "surrogate model"?

What is Bayesian optimization? Introduction Bayesian optimization (BO) is an optimization technique used to model an unknown (usually continuous) function $f: \mathbb{R}^d \rightarrow Y$, where ...
user avatar
  • 33.8k
5 votes
Accepted

Why do we need both the validation set and test set?

The difference between the validation and test set in my opinion should be explained in this way: the validation set is meant to be used multiple times. the test set is meant to be used only once. ...
user avatar
5 votes
Accepted

What is the meaning of "exploration" in reinforcement and supervised learning?

In reinforcement learning, exploration has a specific meaning, which is in contrast with the meaning of exploitation, hence the so-called exploration-exploitation dilemma (or trade-off). You explore ...
user avatar
  • 33.8k
4 votes

How to select number of hidden layers and number of memory cells in an LSTM?

Have a look at the paper Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling (2014), where different LSTM architectures are compared. In the abstract, the ...
user avatar
  • 269
4 votes
Accepted

In Q-learning, shouldn't the learning rate change dynamically during the learning phase?

Yes you can decay the learning rate in Q-learning, and yes this should result in more accurate Q-values in the long term for many environments. However, this is something that is harder to manage ...
user avatar
4 votes
Accepted

How can I avoid overfitting when doing parameter tuning?

Yes. Usually you would use cross validation to avoid overfitting during parameter tuning. If your dataset is large enough, and you don't try too many parameter combinations, this will work well, ...
user avatar
4 votes
Accepted

Is there a reason to choose regular momentum over Nesterov momentum for neural networks?

The book Deep Learning by Goodfellow, Bengio, and Courville says (Sec 8.3.3, p 292 in my copy) states that Unfortunately, in the stochastic gradient case, Nesterov momentum does not improve the ...
user avatar
4 votes
Accepted

How do I design the network for Deep Q-Network?

What is the strategy to get to a better network? There are a few different strategies that you can use to search for good hyperparameters in reinforcement learning RL, but you should be aware that ...
user avatar
3 votes

How should we choose the dimensions of the encoding layer in auto-encoders?

The number of dimensions is a hyperparameter of your model, and you should do a hyperparameter search, like with any other parameters. There's also a tradeoff between dimension and training speed, so ...
user avatar
3 votes
Accepted

What is the pros and cons of increasing and decreasing the number of worker processes in A3C?

The correct number of child processes will depend on the hardware available to you. Simplifying a bit, child processes can be in one of two states: waiting for memory or disk access, or running. If ...
user avatar
3 votes

How to choose an activation function for the hidden layers?

***Take my answer as a side note to that given by cantordust: If one can verify that an activation function perform well in some cases, that good behavior often extrapolates to other problems. Thus, ...
user avatar
  • 31
3 votes

Do genetic algorithms also evolve?

In principle, yes, you can also evolve the genetic algorithm (or, in general, evolutionary algorithm), i.e. you can evolve its operations (such as the mutation and cross-over) and hyper-parameters (...
user avatar
  • 33.8k
3 votes
Accepted

How do I design a neural network that breaks a 5-letter word into its corresponding syllables?

I would highly recommend modeling things differently with regard to how letters are presented to the model. While the problem is more natural, perhaps, for a Convolutional or Recurrent Neural Network, ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible