Questions tagged [hyper-parameters]

For questions related to the hyper-parameters of AI models and algorithms, which are parameters that are set before the learning process begins. For example, the number of hidden layers in a feed-forward neural network is usually a hyper-parameter.

Filter by
Sorted by
Tagged with
57 votes
4 answers
94k views

How to select number of hidden layers and number of memory cells in an LSTM?

I am trying to find some existing research on how to select the number of hidden layers and the size of these of an LSTM-based RNN. Is there an article where this problem is being investigated, i.e., ...
user avatar
33 votes
4 answers
1k views

How to find the optimal number of neurons per layer?

When you're writing your algorithm, how do you know how many neurons you need per single layer? Are there any methods for finding the optimal number of them, or is it a rule of thumb?
user avatar
  • 10k
22 votes
3 answers
10k views

How to choose an activation function for the hidden layers?

I choose the activation function for the output layer depending on the output that I need and the properties of the activation function that I know. For example, I choose the sigmoid function when I'm ...
user avatar
8 votes
2 answers
8k views

How large should the replay buffer be?

I'm learning DDPG algorithm by following the following link: Open AI Spinning Up document on DDPG, where it is written In order for the algorithm to have stable behavior, the replay buffer should ...
user avatar
8 votes
2 answers
4k views

Why should the number of neurons in a hidden layer be a power of 2?

I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster. Is ...
user avatar
  • 205
8 votes
1 answer
544 views

What causes a model to require a low learning rate?

I've pondered this for a while without developing an intuition for the math behind the cause of this. So what causes a model to need a low learning rate?
user avatar
  • 207
7 votes
2 answers
6k views

How do we choose the kernel size depending on the problem?

Obviously, finding suitable hyper-parameters for a neural network is a complex task and problem or domain-specific. However, there should be at least some "rules" that hold most times for the size of ...
user avatar
  • 256
6 votes
1 answer
2k views

Is this idea to calculate the required number of hidden neurons for a single hidden layer neural network correct?

I have an idea to find the optimal number of hidden neurons required in a neural network, but I'm not sure how accurate it is. Assuming that it has only 1 hidden layer, it is a classification problem ...
user avatar
  • 205
6 votes
1 answer
2k views

Should I be decaying the learning rate and the exploration rate in the same manner?

Should I be decaying the learning rate and the exploration rate in the same manner? What's too slow and too fast of an exploration and learning rate decay? Or is it specific from model to model?
user avatar
  • 91
6 votes
1 answer
909 views

How should we choose the dimensions of the encoding layer in auto-encoders?

How should we choose the dimensions of the encoding layer in auto-encoders?
user avatar
  • 101
5 votes
1 answer
314 views

How do I design a neural network that breaks a 5-letter word into its corresponding syllables?

I am going to design a neural network which will be able to break a 5-letter word into its corresponding syllables (hybrid syllables, I mean it will not strictly adhere to grammatical syllable rules ...
user avatar
4 votes
1 answer
67 views

When using Neural Architecture Search, how are the hyper-parameters chosen?

I have read a lot about NAS, but I still do not understand one concept: When setting up a neural network, hyperparameters (such as the learning rate, dropout rate, batch size, filter size, etc.) need ...
user avatar
4 votes
1 answer
863 views

How does L2 regularization make weights smaller?

I'm learning logistic regression and $L_2$ regularization. The cost function looks like below. $$J(w) = -\displaystyle\sum_{i=1}^{n} (y^{(i)}\log(\phi(z^{(i)})+(1-y^{(i)})\log(1-\phi(z^{(i)})))$$ And ...
user avatar
4 votes
1 answer
564 views

What made your DDPG implementation on your environment work?

I am working on scheduling problem that has inherent randomness. The dimensions of action and state spaces are 1 and 5 respectively. I am using DDPG, but it seems extremely unstable, and so far it ...
user avatar
  • 242
4 votes
1 answer
933 views

How many weights does the max-pooling layer have?

How many weights does the max-pooling layer have? For example, if there are 10 inputs, a pooling filter of size 2, stride 2, how many weights, including bias, does a max-pooling layer have?
user avatar
  • 43
4 votes
2 answers
146 views

In a neural network, by how much does the number of neurons typically vary from layer to layer?

In a neural network, by how much does the number of neurons typically vary from layer to layer? Note that I am NOT asking how to find the optimal number of neurons per layer. As a hardware design ...
user avatar
4 votes
1 answer
70 views

How do you efficiently choose the hyper-parameters of a neural network?

How do you efficiently choose the hyper-parameters of a neural network (e.g. the learning rate, number of layer, weights, etc.)?
user avatar
3 votes
2 answers
422 views

When can I call an entity a hyperparameter?

As per my knowledge, any entity that is learnable by a training algorithm can be called a parameter. Weights of a neural network are called parameters because of this reason only. But I have doubts ...
user avatar
  • 3,099
3 votes
2 answers
276 views

What are the best hyper-parameters to tune in reinforcement learning?

Obviously, this is somewhat subjective, but what hyper-parameters typically have the most significant impact on an RL agent's ability to learn? For example, the replay buffer size, learning rate, ...
user avatar
3 votes
2 answers
87 views

Does this hyperparameter optimisation approach yield the optimal hyperparameters?

Say I have a ML model which is not very costly to train. It has around say 5 hyperparameters. One way to select best hyperparameters would be to keep all the other hyperparamaters fixed and train ...
user avatar
3 votes
2 answers
2k views

How to determine the embedding size?

When we are training a neural network, we are going to determine the embedding size to convert the categorical (in NLP, for instance) or continuous (in computer vision or voice) information to hidden ...
user avatar
3 votes
1 answer
77 views

How are training hyperparameters determined for large models?

When training a relatively small DL model, which takes several hours to train, I typically start with some starting points from literature and then use a trial-and-error or grid-search approach to ...
user avatar
  • 133
3 votes
1 answer
424 views

Why does every neuron in hidden layers of a multi-layer perceptron typically have the same activation function? [duplicate]

Why does every neuron in a hidden layer of a multi-layer perceptron (MLP) typically have the same activation function as every other neuron in the same or other hidden layers (so I exclude the output ...
user avatar
3 votes
1 answer
121 views

Why is the number of output channels 16 in the hidden layer of this CNN?

In this tutorial from Jeremy Howard: What is torch.nn really? he has an example towards the end where he creates a CNN for MNIST. In nn.Conv2d, he makes the ...
user avatar
  • 173
3 votes
1 answer
119 views

What is the pros and cons of increasing and decreasing the number of worker processes in A3C?

In A3C, there are several child processes and one master process. The child precesses calculate the loss and backpropagation, and the master process sums them up and updates the parameters, if I ...
user avatar
  • 977
3 votes
1 answer
53 views

Why is a large replay buffer inefficient?

Open AI spin up says ... the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything. If you only use the very-most recent data, ...
user avatar
  • 131
3 votes
3 answers
2k views

Why must the momentum factor be in the range 0-1?

Why is it a bad idea to have a momentum factor greater than 1? What are the mathematical motivations/reasons?
user avatar
3 votes
1 answer
66 views

What is the most statistically acceptable method for tuning neural network hyperparameters on very small datasets?

Neural networks are usually evaluated by dividing a dataset into three splits: training, validation, and test The idea is that critical hyperparameters of the network such as the number of epochs ...
user avatar
  • 391
3 votes
1 answer
1k views

How should I choose the target's update frequency in DQN?

I have been dealing with a problem that I'm trying to solve with DQN. A general question that I have is regarding the target's update frequency. How should it change? Depending on what factor do we ...
user avatar
3 votes
1 answer
1k views

Why can't my implementation of the Actor-Critic algorithm achieve good results in the 2048 game?

I implemented the Actor-Critic with n-step TD prediction to learn to play the 2048 game For the environment, I don't use this 2048 implementation. I use a simple one without any graphical interface, ...
user avatar
3 votes
0 answers
68 views

How can I do hyperparameter optimization for a CNN-LSTM neural network?

I have built a CNN-LSTM neural network with 2 inputs and 2 outputs in Keras. I trained the network with model.fit_generator() (and not ...
user avatar
  • 31
3 votes
0 answers
27 views

How to organize model training hyperparameters

I am working on multiple deep learning projects, most of them in the area of computer vision. For many of them I create multiple models, try different approaches, use various model architectures. And ...
user avatar
  • 155
3 votes
0 answers
34 views

Which hyper-parameters are considered in neural architecture search?

I want to understand automatic Neural Architecture Search (NAS). I read already multiple papers, but I cannot figure out what the actual search space of NAS is / how are classical hyper-parameters ...
user avatar
2 votes
1 answer
110 views

Which hyperparameters in neural network are accesible to users adjustment

I am new to Neural Networks and my questions are still very basic. I know that most of neural networks allow and even ask user to chose hyper-parameters like: amount of hidden layers amount of ...
user avatar
  • 131
2 votes
2 answers
683 views

What is the purpose of the "gamma" parameter in SVMs?

I want to understand what the gamma parameter does in an SVM. According to this page. Intuitively, the gamma parameter defines ...
user avatar
  • 121
2 votes
1 answer
112 views

Is a calculus or ML approach to varying learning rate as a function of loss and epoch been investigated?

Many have examined the idea of modifying learning rate at discrete times during the training of an artificial network using conventional back propagation. The goals of such work have been a balance ...
user avatar
2 votes
1 answer
36 views

For continuing tasks, is the choice of episode length completely arbitrary?

Let's say I'm training a reinforcement learning agent to act in some environment that perpetually continues to give the agent opportunities to earn rewards, and there is no cap on the score and there ...
user avatar
2 votes
1 answer
116 views

Why is the input layer of a neural network usually not counted?

I came across the following statement from the caption of figure 7.8 from the textbook Neural Networks and Neural Language Models the input layer is usually not counted when enumerating layers Why ...
user avatar
  • 3,099
2 votes
1 answer
266 views

Is it a good idea to use different width and height of the kernel in a CNN?

I always see that the width and height of the kernel are the same. But is it a good idea to use different numbers? Recently I tried to use GoogLeNet (which expects images to be 224x224) on my images (...
user avatar
  • 183
2 votes
1 answer
784 views

What should the value of epsilon be in the Q-learning?

I am trying to understand Reinforcement Learning and already explored different Youtube videos, blog posts, and Wikipedia articles. What I don't understand is the impact of $\epsilon$. What value ...
user avatar
2 votes
1 answer
80 views

What components of reinforcement learning influence the result the most?

I'm working on my thesis concerning a reinforcement learning problem and am trying to prioritise my time on different components of it: Formalising the agent environment (like the design of state-, ...
user avatar
  • 85
2 votes
1 answer
319 views

What is the intuition behind the number of filters/channels for each convolutional layer?

After having chosen the number of layers for a convolutional neural network, we must also choose the number of filters/channels for each convolutional layer. The intuition behind the filter's spatial ...
user avatar
  • 23
2 votes
1 answer
50 views

How exactly does nested cross-validation work?

I have trouble understanding how nested cross-validation works - I understand the need for two loops (one for selecting the model, and another for training the selected model), but why are they nested?...
user avatar
2 votes
1 answer
97 views

Can we automate the choice of the hyper-parameters of the evolutionary algorithms?

Certain hyper-parameters (e.g. the size of the offspring generation or the definition of the fitness function) and the design (e.g. how the mutation is performed) of evolutionary algorithms usually ...
user avatar
  • 331
2 votes
1 answer
428 views

Maximum number of neurons in a layer given number of neurons in previous layer

Consider an extremely complicated feed-forward neural network training example but with no need of computational efficiency or limiting of processing time. What is the maximum number of hidden ...
user avatar
2 votes
1 answer
527 views

How to design a neural network to predict the quadrant where a given point lies?

I've written a single perceptron that can predict whether a point is above or below a straight-line graph, given the correct training data and using a sign activation function. Now, I'm trying to ...
user avatar
  • 205
2 votes
0 answers
779 views

Optimal episode length in reinforcement learning

I have a custom environment for stock trading where an episode can be as long as 2000-3000 steps. I've run several experiments with td3 and sac algorithms, average reward per episode flattens after ...
user avatar
  • 167
2 votes
0 answers
29 views

Is there an optimal number of species for NEAT?

Is there an optimal number of species for NEAT? Since too low and too high is bad, I am thinking about adjusting the threshold of the distance function at runtime in order to have the number of ...
user avatar
  • 51
2 votes
0 answers
42 views

Given a 2-layer GCN, can we choose the dimensions of the 2nd weight matrix, such that this architecture has the same capacity as a 1-layer GCN?

This might be more of a question about nested function classes: For $k$ class node classification in a graph with $n$ nodes, and $d$ feature vector. I want to compare Architecture I: the GCN model of ...
user avatar
  • 121
2 votes
0 answers
72 views

How will the filter size affect the transpose convolution operation?

After a series of convolutions, I am up-sampling a compressed representation, I was curious what is the methodology I should follow to choose an optimum kernel size for up-sampling. How will the ...
user avatar
  • 31