# Tag Info

### How to select number of hidden layers and number of memory cells in an LSTM?

Your question is quite broad, but here are some tips. Specifically for LSTMs, see this Reddit discussion Does the number of layers in an LSTM network affect its ability to remember long patterns? ...
• 1,117
Accepted

### How to find the optimal number of neurons per layer?

There is no direct way to find the optimal number of them: people empirically try and see (e.g., using cross-validation). The most common search techniques are random, manual, and grid searches. ...
• 2,036
Accepted

### How to choose an activation function for the hidden layers?

It seems to me that you already understand the shortcomings of ReLUs and sigmoids (like dead neurons in the case of plain ReLU). You may want to look at ELU (exponential linear units) and SELU (self-...
• 923
Accepted

### Why should the number of neurons in a hidden layer be a power of 2?

I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster. I ...
• 24k
Accepted

### How can I automate the choice of the architecture of a neural network for an arbitrary problem?

I think in this case, you'll probably want to use a genetic algorithm to generate a topology rather than working on your own. I personally like NEAT (NeuroEvolution of Augmenting Topologies). The ...
• 226

### How to select number of hidden layers and number of memory cells in an LSTM?

The selection of the number of hidden layers and the number of memory cells in LSTM probably depends on the application domain and context where you want to apply this LSTM. The optimal number of ...

### How to select number of hidden layers and number of memory cells in an LSTM?

In general, there are no guidelines on how to determine the number of layers or the number of memory cells in an LSTM. The number of layers and cells required in an LSTM might depend on several ...
• 699

### How to find the optimal number of neurons per layer?

For a more intelligent approach than random or exhaustive searches, you could try a genetic algorithm such as NEAT http://nn.cs.utexas.edu/?neat. However, this has no guarantee to find a global optima,...
• 702
Accepted

### How do I decide the optimal number of layers for a neural network?

There is a technique called Pruning in neural networks, which is used just for this same purpose. The pruning is done on the number of hidden layers. The process ...
• 1,351
Accepted

### Is this idea to calculate the required number of hidden neurons for a single hidden layer neural network correct?

I have an idea to find the optimal number of hidden neurons required in a neural network but I'm not sure how accurate it is. It's a complete non-starter, and there is a no such calculation possible ...
• 24k

### How to find the optimal number of neurons per layer?

Paper Szegedy C, Vanhoucke V, Ioffe S, et al. Rethinking the inception architecture for computer vision[J]. arXiv preprint arXiv:1512.00567, 2015. gives some general design principles: Avoid ...
• 161

### How is neural architecture search performed?

You could say that NAS fits into the domain of Meta Learning or Meta Machine learning. I've pulled the NAS papers from my notes, this is a collection of papers/lectures that I personally found very ...
Accepted

### Do we need automatic hyper-parameter tuning when we have a large enough dataset?

Unfortunately, even with large amounts of training data, hyperparameter choices can strongly influence the performance of a trained model. What you can usually drop when you have large amounts of ...
• 24k
Accepted

### How to shorten the development time of a neural network?

Your scenario is common. The most straightforward approach is to subsample your data randomly. Unless your data or your model has strong bias, your performance to the smaller data set should be ...
• 1,390

### How can I automate the choice of the architecture of a neural network for an arbitrary problem?

The other answer mentions NEAT to generate network weights or topologies. The paper NeuroEvolution: The Importance of Transfer Function Evolution and Heterogeneous Networks, which also gives a short ...
• 7,046

### What is a "surrogate model"?

A surrogate model is a simplified model. It is a mapping $y_S=f_S(x)$ that approximates the original model $y=f(x)$, in a given domain, reasonably well. Source: Engineering Design via Surrogate ...
• 171
Accepted

### What is a "surrogate model"?

What is Bayesian optimization? Introduction Bayesian optimization (BO) is an optimization technique used to model an unknown (usually continuous) function $f: \mathbb{R}^d \rightarrow Y$, where ...
• 33.8k
Accepted

### Why do we need both the validation set and test set?

The difference between the validation and test set in my opinion should be explained in this way: the validation set is meant to be used multiple times. the test set is meant to be used only once. ...
• 3,528
Accepted

### What is the meaning of "exploration" in reinforcement and supervised learning?

In reinforcement learning, exploration has a specific meaning, which is in contrast with the meaning of exploitation, hence the so-called exploration-exploitation dilemma (or trade-off). You explore ...
• 33.8k

### How to select number of hidden layers and number of memory cells in an LSTM?

Have a look at the paper Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling (2014), where different LSTM architectures are compared. In the abstract, the ...
• 269
Accepted

### In Q-learning, shouldn't the learning rate change dynamically during the learning phase?

Yes you can decay the learning rate in Q-learning, and yes this should result in more accurate Q-values in the long term for many environments. However, this is something that is harder to manage ...
• 24k
Accepted

### How can I avoid overfitting when doing parameter tuning?

Yes. Usually you would use cross validation to avoid overfitting during parameter tuning. If your dataset is large enough, and you don't try too many parameter combinations, this will work well, ...
• 8,877
Accepted

### Is there a reason to choose regular momentum over Nesterov momentum for neural networks?

The book Deep Learning by Goodfellow, Bengio, and Courville says (Sec 8.3.3, p 292 in my copy) states that Unfortunately, in the stochastic gradient case, Nesterov momentum does not improve the ...
• 124
Accepted

### How do I design the network for Deep Q-Network?

What is the strategy to get to a better network? There are a few different strategies that you can use to search for good hyperparameters in reinforcement learning RL, but you should be aware that ...
• 24k

### How should we choose the dimensions of the encoding layer in auto-encoders?

The number of dimensions is a hyperparameter of your model, and you should do a hyperparameter search, like with any other parameters. There's also a tradeoff between dimension and training speed, so ...
Accepted

### What is the pros and cons of increasing and decreasing the number of worker processes in A3C?

The correct number of child processes will depend on the hardware available to you. Simplifying a bit, child processes can be in one of two states: waiting for memory or disk access, or running. If ...
• 8,877

### How to choose an activation function for the hidden layers?

***Take my answer as a side note to that given by cantordust: If one can verify that an activation function perform well in some cases, that good behavior often extrapolates to other problems. Thus, ...
• 31