Questions tagged [hyper-parameters]

For questions related to the hyper-parameters of AI models and algorithms, which are parameters that are set before the learning process begins. For example, the number of hidden layers in a feed-forward neural network is usually a hyper-parameter.

Filter by
Sorted by
Tagged with
2
votes
1answer
77 views

What made your DDPG implementation on your environment work?

I am working on scheduling problem that has inherent randomness. The dimensions of action and state spaces are 1 and 5 respectively. I am using DDPG, but it seems extremely unstable, and so far it ...
1
vote
0answers
19 views

Should we start with a small batch-size and increase during training to improve sample efficiency?

Just made an interesting observation playing around with the stable-baseline's implementation of PPO and the BipedalWalker environment from OpenAI's Gym. But I believe this should be a general ...
1
vote
1answer
18 views

How to know if the hyperparameters of a neural network relate to each other?

According this thread some hyperparameters are independent from each other while some are directly related. One of the answers give an example where two hyperparameters affect each other. For ...
1
vote
0answers
36 views

Why is the number of neurons used in various neural networks power of 2?

I have noticed that almost all tutorials take the number of neurons as a power of 2. Is there any proper mathematical and well-proven reason for that? If you sometimes change it to some other odd ...
1
vote
1answer
45 views

What is relation between gradient descent and regularization in deep learning?

Gradient descent is used to reduce the loss and regularization is used to fight over-fitting. Is there any relation between gradient descent and regularization, or both are independent of each other?...
1
vote
0answers
49 views

Why do momentum techniques not work well for RNNs?

AFAIK, momentum is quite useful when training CNNs, and can speed-up the training substantially without any drop in validation accuracy. I've recently learned that it is not as helpful for RNNs ...
3
votes
0answers
34 views

How can I do hyperparameter optimization for a CNN-LSTM neural network?

I have built a CNN-LSTM neural network with 2 inputs and 2 outputs in Keras. I trained the network with model.fit_generator() (and not ...
2
votes
0answers
16 views

Which CNN hyper-parameters are most sensitive to centered versus off centered data?

Which hyper-parameters of a convolutional neural network are likely to be the most sensitive to depending on whether the training (and test and inference) data involves only accurately centered images ...
3
votes
0answers
17 views

How to organize model training hyperparameters

I am working on multiple deep learning projects, most of them in the area of computer vision. For many of them I create multiple models, try different approaches, use various model architectures. And ...
2
votes
1answer
35 views

Will a .h5 file trained with Xception model work with Resnet50?

I have been running my 2013 server box since 2 weeks ago for training an AI model. I set up 30 epochs to run but since than it only ran 1 epoch as my PC config is super slow. But it generates 1 .h5 ...
2
votes
2answers
49 views

Does this hyperparameter optimisation approach yield the optimal hyperparameters?

Say I have a ML model which is not very costly to train. It has around say 5 hyperparameters. One way to select best hyperparameters would be to keep all the other hyperparamaters fixed and train ...
2
votes
1answer
73 views

Can we automate the choice of the hyper-parameters of the evolutionary algorithms?

Certain hyper-parameters (e.g. the size of the offspring generation or the definition of the fitness function) and the design (e.g. how the mutation is performed) of evolutionary algorithms usually ...
2
votes
0answers
27 views

What is the benefit of scaling the hyperparameter C of an SVM?

Please read the following page of the Sklearn documentation. The figure shown there (see below) illustrates why C should be scaled when using a SVM with 'l1' penalty, whereas it shouldn't be scaled ...
8
votes
1answer
141 views

What causes a model to require a low learning rate?

I've pondered this for a while without developing an intuition for the math behind the cause of this. So what causes a model to need a low learning rate?
3
votes
1answer
47 views

How do you efficiently choose the hyper-parameters of a neural network?

How do you efficiently choose the hyper-parameters of a neural network (e.g. the learning rate, number of layer, weights, etc.)?
7
votes
1answer
385 views

How should we choose the dimensions of the encoding layer in auto-encoders?

How should we choose the dimensions of the encoding layer in auto-encoders?
2
votes
1answer
150 views

Maximum number of neurons in a layer given number of neurons in previous layer

Consider an extremely complicated feed-forward neural network training example but with no need of computational efficiency or limiting of processing time. What is the maximum number of hidden ...
0
votes
1answer
47 views

How many trees should be generated in a random forest?

What are ways of determining the number of trees to be generated in a random forest algorithm?
2
votes
1answer
93 views

Is a calculus or ML approach to varying learning rate as a function of loss and epoch been investigated?

Many have examined the idea of modifying learning rate at discrete times during the training of an artificial network using conventional back propagation. The goals of such work have been a balance ...
8
votes
2answers
5k views

How do we choose the kernel size depending on the problem?

Obviously, finding suitable hyper-parameters for a neural network is a complex task and problem or domain-specific. However, there should be at least some "rules" that hold most times for the size of ...