Questions tagged [hyperparameter-optimization]

For questions related to the concept of hyper-parameter optimization, that is, the task of finding the best hyper-parameters for a particular learning algorithm (e.g. gradient descent) or model (e.g. a multi-layer neural network) using an optimization method (e.g. Bayesian optimization or genetic algorithms).

Filter by
Sorted by
Tagged with
33
votes
4answers
1k views

How to find the optimal number of neurons per layer?

When you're writing your algorithm, how do you know how many neurons you need per single layer? Are there any methods for finding the optimal number of them, or is it a rule of thumb?
22
votes
3answers
9k views

How to choose an activation function for the hidden layers?

I choose the activation function for the output layer depending on the output that I need and the properties of the activation function that I know. For example, I choose the sigmoid function when I'm ...
17
votes
2answers
296 views

How do I decide the optimal number of layers for a neural network?

How do I decide the optimal number of layers for a neural network (feedforward or recurrent)?
13
votes
1answer
194 views

Will parameter sweeping on one split of data followed by cross validation discover the right hyperparameters?

Let's call our dataset splits train/test/evaluate. We're in a situation where we require months of data. So we prefer to use the evaluation dataset as infrequently as possible to avoid polluting our ...
8
votes
2answers
6k views

How do we choose the kernel size depending on the problem?

Obviously, finding suitable hyper-parameters for a neural network is a complex task and problem or domain-specific. However, there should be at least some "rules" that hold most times for the size of ...
7
votes
2answers
3k views

Why should the number of neurons in a hidden layer be a power of 2?

I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster. Is ...
7
votes
1answer
665 views

How should we choose the dimensions of the encoding layer in auto-encoders?

How should we choose the dimensions of the encoding layer in auto-encoders?
6
votes
1answer
1k views

Is this idea to calculate the required number of hidden neurons for a single hidden layer neural network correct?

I have an idea to find the optimal number of hidden neurons required in a neural network, but I'm not sure how accurate it is. Assuming that it has only 1 hidden layer, it is a classification problem ...
6
votes
2answers
191 views

How to shorten the development time of a neural network?

I am developing an LSTM for sequence tagging. During the development, I do various changes in the system, for example, add new features, change the number of nodes in the hidden layers, etc. After ...
5
votes
3answers
265 views

How is neural architecture search performed?

I have come across something that IBM offers called neural Architecture search. You feed it a data set and it outputs an initial neural Architecture that you can train. How is neural architecture ...
5
votes
1answer
151 views

How can I avoid overfitting when doing parameter tuning?

I very often applied a grid search to tune the parameters of my supervised model. I have the feeling that parameter tuning will eventually (very often) lead to overfitting? Is this crazy to say? Is ...
5
votes
2answers
336 views

Do genetic algorithms also evolve?

After witnessing the rise of deep learning as automatic feature/pattern recognition over classic machine learning techniques, I had an insight that the more you automate at each level, the better the ...
5
votes
1answer
51 views

How can we conclude that an optimization algorithm is better than another one

When we test a new optimization algorithm, what the process that we need to do?For example, do we need to run the algorithm several times, and pick a best performance,i.e., in terms of accuracy, f1 ...
5
votes
1answer
249 views

An intuitive explanation of Adagrad, its purpose and its formula

It (Adagrad) adapts the learning rate to the parameters, performing smaller updates (i.e. low learning rates) for parameters associated with frequently occurring features, and larger updates (i.e. ...
4
votes
3answers
2k views

What is a "surrogate model"?

In the following paragraph from the book Automated Machine Learning: Methods, Systems, Challenges (by Frank Hutter et al.) In this section we first give a brief introduction to Bayesian ...
4
votes
2answers
156 views

Why do we need both the validation set and test set?

I know that this has been asked a hundred times before, however, I was not able to find a question (and an answer) which actually answered what I wanted to know, respectively, which explained it in a ...
4
votes
1answer
139 views

Is there a reason to choose regular momentum over Nesterov momentum for neural networks?

I've been reading about Nesterov momentum from here and it seems like a nice improvement over regular momentum with no extra cost whatsoever. However, is this really the case? Are there instances ...
4
votes
1answer
83 views

Bayesian hyperparameter optimization, is it worth it?

In the Deep Learning book by Goodfellow et al., section 11.4.5 (p. 438), the following claims can be found: Currently, we cannot unambiguously recommend Bayesian hyperparameter optimization as an ...
4
votes
2answers
326 views

What is the meaning of "exploration" in reinforcement and supervised learning?

While exploration is an integral part of reinforcement learning (RL), it does not pertain to supervised learning (SL) since the latter is already provided with the data set from the start. That said, ...
4
votes
1answer
121 views

Are there Python packages for random search hyper-parameter optimisation? [closed]

Which Python packages do you recommend for random search hyperparameter optimization to use? Is there any recent and good one (better than the one in scikit-learn)?
3
votes
2answers
98 views

What are the best hyper-parameters to tune in reinforcement learning?

Obviously, this is somewhat subjective, but what hyper-parameters typically have the most significant impact on an RL agent's ability to learn? For example, the replay buffer size, learning rate, ...
3
votes
1answer
70 views

How are training hyperparameters determined for large models?

When training a relatively small DL model, which takes several hours to train, I typically start with some starting points from literature and then use a trial-and-error or grid-search approach to ...
3
votes
1answer
73 views

What is the pros and cons of increasing and decreasing the number of worker processes in A3C?

In A3C, there are several child processes and one master process. The child precesses calculate the loss and backpropagation, and the master process sums them up and updates the parameters, if I ...
3
votes
1answer
94 views

How to find proper parameter settings for a given optimization algorithm?

Is there any methodology to find proper parameter settings for a given meta-heuristic algorithm, e.g. the firefly algorithm or the cuckoo search? Is this an open issue in optimization? Is extensive ...
3
votes
1answer
635 views

How should I choose the target's update frequency in DQN?

I have been dealing with a problem that I'm trying to solve with DQN. A general question that I have is regarding the target's update frequency. How should it change? Depending on what factor do we ...
3
votes
0answers
83 views

Are there principled ways of tuning a neural network in case of overfitting and underfitting?

Whenever I tune my neural network, I usually take the common approach of defining some layers with some neurons. If it overfits, I reduce the layers, neurons, add dropout, utilize regularisation. ...
3
votes
1answer
253 views

When training a CNN, what are the hyperparameters to tune first?

I am training a convolutional neural network for object detection. Apart from the learning rate, what are the other hyperparameters that I should tune? And in what order of importance? Besides, I read ...
3
votes
0answers
41 views

What are some ways to quickly evaluate the potential of a given NN architecture?

Main question Is there some way we can leverage general knowledge of how certain hyperparameters affect performance, to very rapidly get some sort of estimate for how good a given architecture could ...
3
votes
0answers
56 views

How can I do hyperparameter optimization for a CNN-LSTM neural network?

I have built a CNN-LSTM neural network with 2 inputs and 2 outputs in Keras. I trained the network with model.fit_generator() (and not ...
3
votes
0answers
21 views

How to organize model training hyperparameters

I am working on multiple deep learning projects, most of them in the area of computer vision. For many of them I create multiple models, try different approaches, use various model architectures. And ...
2
votes
2answers
63 views

Does this hyperparameter optimisation approach yield the optimal hyperparameters?

Say I have a ML model which is not very costly to train. It has around say 5 hyperparameters. One way to select best hyperparameters would be to keep all the other hyperparamaters fixed and train ...
2
votes
2answers
73 views

How to determine the embedding size?

When we are training a neural network, we are going to determine the embedding size to convert the categorical (in NLP, for instance) or continuous (in computer vision or voice) information to hidden ...
2
votes
1answer
59 views

What is the impact of changing the crossover and mutation rates?

What is the impact of using a: low crossover rate high crossover rate low mutation rate high mutation rate
2
votes
1answer
54 views

After having selected the best model with cross-validation, for how long should I train it?

When using k-fold cross-validation in a deep learning problem, after you have computed your hyper-parameters, how do you decide how long to train your final model? My understanding is that, after the ...
2
votes
1answer
641 views

How do I choose the size of the hidden state of a GRU?

I'm trying to understand how the size of the hidden state affects the GRU. For example, suppose I want to make a GRU count. I'm gonna feed it with three numbers, and I expect it to predict the ...
2
votes
1answer
50 views

What is the most statistically acceptable method for tuning neural network hyperparameters on very small datasets?

Neural networks are usually evaluated by dividing a dataset into three splits: training, validation, and test The idea is that critical hyperparameters of the network such as the number of epochs ...
2
votes
1answer
50 views

Is it possible to train one part of the network with a particular learning rate and the other part with a different one?

I have a combined network consisting of two parts: one is for images and the other is for numerical data. Each sample is matched with a numerical case by an ID. For this combined network, a ...
2
votes
1answer
104 views

What should the value of epsilon be in the Q-learning?

I am trying to understand Reinforcement Learning and already explored different Youtube videos, blog posts, and Wikipedia articles. What I don't understand is the impact of $\epsilon$. What value ...
2
votes
1answer
55 views

How should I change the hyper-parameters of the C51 algorithm, in order to obtain higher reward?

I have a scenario where, in an ideal situation, the greedy approach is the best, but when non-idealities are introduced which can be learned, DQN starts doing better. So, after checking what DQN ...
2
votes
1answer
86 views

Can we automate the choice of the hyper-parameters of the evolutionary algorithms?

Certain hyper-parameters (e.g. the size of the offspring generation or the definition of the fitness function) and the design (e.g. how the mutation is performed) of evolutionary algorithms usually ...
2
votes
0answers
13 views

Is there an optimal number of species for NEAT?

Is there an optimal number of species for NEAT? Since too low and too high is bad, I am thinking about adjusting the threshold of the distance function at runtime in order to have the number of ...
2
votes
0answers
19 views

How to Select Model Parameters for Transformer (Heads, number of layers, etc)

Is there a general guideline on how the Transformer model parameters should be selected, or the range of these parameters that should be included in a hyperparameter sweep? Number of heads Number of ...
2
votes
0answers
25 views

Deriving hyperparameter updates in Online Interactive Collaborative Filtering

I've been going through "Online Interactive Collaborative Filtering Using Multi-Armed Bandit with Dependent Arms" by Wang et al. and am unable to understand how the update equations for the ...
2
votes
0answers
24 views

Is it normal to see oscillations in tested hyperparameters during bayesian optimisation?

I've been trying out bayesian hyperparameter optimisation (with TPE) on a simple CNN applied to the MNIST handwritten digit dataset. I noticed that over iterations of the optimisation loop, the tested ...
1
vote
2answers
65 views

Why is an embedding of dimension 400 enough to represent 70000 words?

I am learning PyTorch on Udacity. In lesson 8, section 11: Training the Model, the instructor writes: Then I have my embedding and hidden dimension. The embedding dimension is just a smaller ...
1
vote
2answers
413 views

How is a validation set used to tune the hyperparameters in a non-biased way, if the new models depends on the values of these?

I've built a neural network from the scratch, choosing arbitrary numbers for the hyperparameters: learning rate, number of hidden layers and neurons for these, number of epochs and size of mini ...
1
vote
1answer
82 views

How to fairly conduct a model performance with 5-fold cross validation after augmentation?

I have, say, a (balanced) data-set with 2k images for binary classification. What I have done is that randomly divided the data-set into 5 folds; copy-pasted all 5-fold data-set to have 5 exact ...
1
vote
1answer
40 views

Can the optimal learning rate differ for different architectures?

In several courses and tutorials about neural networks, people often say that the learning rate (LR) should be the first hyper-parameter to be tuned before we tweak the others. For example, in this ...
1
vote
1answer
48 views

What is the difference between sensitivity analysis and parameter tuning?

I tried different values of genetic algorithm operators: many crossover rates from 20% to 80% many crossover rates from 1% to 20% varying the population size The study of different parameter values ...
1
vote
1answer
421 views

What is the best activation function for the embedding layer in a deep auto-encoder?

I am designing a deep autoencoder for graph embedding (exactly node embedding) following this paper SDNE. In the original paper, they used the sigmoid activation for all hidden layers in the ...