Questions tagged [hyperparameter-optimization]
For questions related to the concept of hyper-parameter optimization, that is, the task of finding the best hyper-parameters for a particular learning algorithm (e.g. gradient descent) or model (e.g. a multi-layer neural network) using an optimization method (e.g. Bayesian optimization or genetic algorithms).
119
questions
65
votes
4
answers
116k
views
How to select number of hidden layers and number of memory cells in an LSTM?
I am trying to find some existing research on how to select the number of hidden layers and the size of these of an LSTM-based RNN.
Is there an article where this problem is being investigated, i.e., ...
- 979
33
votes
4
answers
1k
views
How to find the optimal number of neurons per layer?
When you're writing your algorithm, how do you know how many neurons you need per single layer? Are there any methods for finding the optimal number of them, or is it a rule of thumb?
- 10.4k
24
votes
3
answers
12k
views
How to choose an activation function for the hidden layers?
I choose the activation function for the output layer depending on the output that I need and the properties of the activation function that I know. For example, I choose the sigmoid function when I'm ...
- 696
18
votes
2
answers
479
views
How do I decide the optimal number of layers for a neural network?
How do I decide the optimal number of layers for a neural network (feedforward or recurrent)?
- 283
16
votes
2
answers
732
views
How can I automate the choice of the architecture of a neural network for an arbitrary problem?
Assume that I want to solve an issue with a neural network that either I can't fit to existing architectures (perceptron, Konohen, etc) or I'm simply not aware of the existence of those or I'm unable ...
- 633
16
votes
1
answer
350
views
Will parameter sweeping on one split of data followed by cross validation discover the right hyperparameters?
Let's call our dataset splits train/test/evaluate. We're in a situation where we require months of data. So we prefer to use the evaluation dataset as infrequently as possible to avoid polluting our ...
- 161
8
votes
2
answers
7k
views
Why should the number of neurons in a hidden layer be a power of 2?
I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster.
Is ...
- 205
8
votes
3
answers
16k
views
How to determine the embedding size?
When we are training a neural network, we are going to determine the embedding size to convert the categorical (in NLP, for instance) or continuous (in computer vision or voice) information to hidden ...
- 897
8
votes
1
answer
478
views
An intuitive explanation of Adagrad, its purpose and its formula
It (Adagrad) adapts the learning rate to the parameters, performing smaller updates
(i.e. low learning rates) for parameters associated with frequently occurring features, and larger updates (i.e. ...
- 123
7
votes
2
answers
7k
views
How do we choose the kernel size depending on the problem?
Obviously, finding suitable hyper-parameters for a neural network is a complex task and problem or domain-specific. However, there should be at least some "rules" that hold most times for the size of ...
- 256
7
votes
1
answer
3k
views
How do we decide which membership function to use?
In classical set theory, there are two options for an element. It is either a member of a set or not. But in fuzzy set theory, there are membership functions to define the "rate" of an ...
- 89
7
votes
3
answers
2k
views
When training a CNN, what are the hyperparameters to tune first?
I am training a convolutional neural network for object detection. Apart from the learning rate, what are the other hyperparameters that I should tune? And in what order of importance? Besides, I read ...
- 71
6
votes
3
answers
3k
views
What is a "surrogate model"?
In the following paragraph from the book Automated Machine Learning: Methods, Systems, Challenges (by Frank Hutter et al.)
In this section we first give a brief introduction to Bayesian ...
- 163
6
votes
1
answer
2k
views
Is this idea to calculate the required number of hidden neurons for a single hidden layer neural network correct?
I have an idea to find the optimal number of hidden neurons required in a neural network, but I'm not sure how accurate it is.
Assuming that it has only 1 hidden layer, it is a classification problem ...
- 205
6
votes
2
answers
231
views
How to shorten the development time of a neural network?
I am developing an LSTM for sequence tagging. During the development, I do various changes in the system, for example, add new features, change the number of nodes in the hidden layers, etc. After ...
6
votes
1
answer
2k
views
How should we choose the dimensions of the encoding layer in auto-encoders?
How should we choose the dimensions of the encoding layer in auto-encoders?
- 101
5
votes
2
answers
757
views
Do we need automatic hyper-parameter tuning when we have a large enough dataset?
Hyperparameter tuning is the process of selecting the optimal hyperparameters for an ANN.
Now, my guess is that, if we have sufficient data (say, 1.4 million for, say, 6 features), the model can be ...
- 299
5
votes
2
answers
821
views
Why do we need both the validation set and test set?
I know that this has been asked a hundred times before, however, I was not able to find a question (and an answer) which actually answered what I wanted to know, respectively, which explained it in a ...
- 173
5
votes
2
answers
3k
views
What are the best hyper-parameters to tune in reinforcement learning?
Obviously, this is somewhat subjective, but what hyper-parameters typically have the most significant impact on an RL agent's ability to learn? For example, the replay buffer size, learning rate, ...
- 243
5
votes
3
answers
350
views
How is neural architecture search performed?
I have come across something that IBM offers called neural architecture search. You feed it a data set and it outputs an initial neural architecture that you can train.
How is neural architecture ...
5
votes
1
answer
440
views
How can I avoid overfitting when doing parameter tuning?
I very often applied a grid search to tune the parameters of my supervised model. I have the feeling that parameter tuning will eventually (very often) lead to overfitting? Is this crazy to say?
Is ...
- 579
5
votes
2
answers
360
views
Do genetic algorithms also evolve?
After witnessing the rise of deep learning as automatic feature/pattern recognition over classic machine learning techniques, I had an insight that the more you automate at each level, the better the ...
5
votes
1
answer
353
views
How do I design a neural network that breaks a 5-letter word into its corresponding syllables?
I am going to design a neural network which will be able to break a 5-letter word into its corresponding syllables (hybrid syllables, I mean it will not strictly adhere to grammatical syllable rules ...
- 164
5
votes
1
answer
78
views
How can we conclude that an optimization algorithm is better than another one
When we test a new optimization algorithm, what the process that we need to do?For example, do we need to run the algorithm several times, and pick a best performance,i.e., in terms of accuracy, f1 ...
- 51
4
votes
1
answer
450
views
Is there a reason to choose regular momentum over Nesterov momentum for neural networks?
I've been reading about Nesterov momentum from here and it seems like a nice improvement over regular momentum with no extra cost whatsoever.
However, is this really the case? Are there instances ...
- 980
4
votes
1
answer
139
views
Bayesian hyperparameter optimization, is it worth it?
In the Deep Learning book by Goodfellow et al., section 11.4.5 (p. 438), the following claims can be found:
Currently, we cannot unambiguously recommend Bayesian hyperparameter
optimization as an ...
- 183
4
votes
2
answers
1k
views
What is the meaning of "exploration" in reinforcement and supervised learning?
While exploration is an integral part of reinforcement learning (RL), it does not pertain to supervised learning (SL) since the latter is already provided with the data set from the start.
That said, ...
- 177
4
votes
1
answer
85
views
How do you efficiently choose the hyper-parameters of a neural network?
How do you efficiently choose the hyper-parameters of a neural network (e.g. the learning rate, number of layer, weights, etc.)?
- 41
3
votes
1
answer
3k
views
In Q-learning, shouldn't the learning rate change dynamically during the learning phase?
I have the following code (below), where an agent uses Q-learning (RL) to play a simple game.
What appears to be questionable for me in that code is the fixed learning rate. When it's set low, it's ...
- 279
3
votes
1
answer
411
views
How do I design the network for Deep Q-Network?
I am playing with deep q-learning and I am thinking about what the proper architecture of a network used for deep q-learning is.
I have a very simple environment, basically a 18x18 matrix, where 3 ...
- 133
3
votes
2
answers
122
views
Does this hyperparameter optimisation approach yield the optimal hyperparameters?
Say I have a ML model which is not very costly to train. It has around say 5 hyperparameters.
One way to select best hyperparameters would be to keep all the other hyperparamaters fixed and train ...
user9947
3
votes
1
answer
202
views
How should I change the hyper-parameters of the C51 algorithm, in order to obtain higher reward?
I have a scenario where, in an ideal situation, the greedy approach is the best, but when non-idealities are introduced which can be learned, DQN starts doing better. So, after checking what DQN ...
- 185
3
votes
1
answer
123
views
How are training hyperparameters determined for large models?
When training a relatively small DL model, which takes several hours to train, I typically start with some starting points from literature and then use a trial-and-error or grid-search approach to ...
- 133
3
votes
1
answer
2k
views
How do I choose the size of the hidden state of a GRU?
I'm trying to understand how the size of the hidden state affects the GRU.
For example, suppose I want to make a GRU count. I'm gonna feed it with three numbers, and I expect it to predict the ...
- 1,118
3
votes
1
answer
194
views
What is the pros and cons of increasing and decreasing the number of worker processes in A3C?
In A3C, there are several child processes and one master process. The child precesses calculate the loss and backpropagation, and the master process sums them up and updates the parameters, if I ...
- 1,037
3
votes
1
answer
3k
views
What should the value of epsilon be in the Q-learning?
I am trying to understand Reinforcement Learning and already explored different Youtube videos, blog posts, and Wikipedia articles.
What I don't understand is the impact of $\epsilon$. What value ...
- 293
3
votes
1
answer
112
views
How to find proper parameter settings for a given optimization algorithm?
Is there any methodology to find proper parameter settings for a given meta-heuristic algorithm, e.g. the firefly algorithm or the cuckoo search? Is this an open issue in optimization? Is extensive ...
- 91
3
votes
1
answer
109
views
What is the most statistically acceptable method for tuning neural network hyperparameters on very small datasets?
Neural networks are usually evaluated by dividing a dataset into three splits:
training,
validation, and
test
The idea is that critical hyperparameters of the network such as the number of epochs ...
- 401
3
votes
1
answer
2k
views
How should I choose the target's update frequency in DQN?
I have been dealing with a problem that I'm trying to solve with DQN. A general question that I have is regarding the target's update frequency. How should it change? Depending on what factor do we ...
3
votes
0
answers
30
views
How should I compare multiple machine learning models to be (generally) fair to all models?
I am testing multiple models on IBM HR Analytics Attrition Dataset (1470 lines) and HR Analytics dataset (15000 lines) for a research project. The models include traditional models (Naive Bayes, SVM), ...
- 131
3
votes
1
answer
261
views
How do I choose the hyper-parameters for a model to detect different guitar chords?
I need to build a hand detector that recognizes the chord played by a hand on a guitar.
I read this article Static Hand Gesture Recognition using Convolutional Neural Network with Data Augmentation ...
- 83
3
votes
0
answers
476
views
How to Select Model Parameters for Transformer (Heads, number of layers, etc)
Is there a general guideline on how the Transformer model parameters should be selected, or the range of these parameters that should be included in a hyperparameter sweep?
Number of heads
Number of ...
- 331
3
votes
0
answers
94
views
Are there principled ways of tuning a neural network in case of overfitting and underfitting?
Whenever I tune my neural network, I usually take the common approach of defining some layers with some neurons.
If it overfits, I reduce the layers, neurons, add dropout, utilize regularisation.
...
- 151
3
votes
0
answers
49
views
What are some ways to quickly evaluate the potential of a given NN architecture?
Main question
Is there some way we can leverage general knowledge of how certain hyperparameters affect performance, to very rapidly get some sort of estimate for how good a given architecture could ...
- 1,319
3
votes
0
answers
157
views
How can I do hyperparameter optimization for a CNN-LSTM neural network?
I have built a CNN-LSTM neural network with 2 inputs and 2 outputs in Keras. I trained the network with model.fit_generator() (and not ...
- 31
3
votes
0
answers
33
views
How to organize model training hyperparameters
I am working on multiple deep learning projects, most of them in the area of computer vision. For many of them I create multiple models, try different approaches, use various model architectures. And ...
- 165
2
votes
3
answers
15k
views
What kind of optimizer is suggested to use for binary classification of similar images?
I have spent some time searching Google and wasn't able to find out what kind of optimization algorithm is best for binary classification when images are similar to one another.
I'd like to read ...
- 241
2
votes
3
answers
225
views
Is it possible to learn the number of layers?
Is it possible, in a transformer or other deep architecture, to include the number of layers as a parameter of the model so it could be learned?
In fact, I have a keras layer that I use to change the ...
- 51
2
votes
1
answer
1k
views
What is the impact of changing the crossover and mutation rates?
What is the impact of using a:
low crossover rate
high crossover rate
low mutation rate
high mutation rate
- 131
2
votes
1
answer
129
views
What is the basic difference between NAS, Hyperparameter Optimization/Tuning and Pruning?
As far as I understand:
NAS = an algorithm that searches for the best NN architecture, i.e., how many layers, what activation function to use, how many neurons, etc.
Hyperparameter Optim = finding ...
- 299