Questions tagged [hyper-parameters]

For questions related to the hyper-parameters of AI models and algorithms, which are parameters that are set before the learning process begins. For example, the number of hidden layers in a feed-forward neural network is usually a hyper-parameter.

Filter by
Sorted by
Tagged with
0
votes
0answers
22 views

Is there any way to determine/estimate the number of rounds for the whole Federated Learning process?

In Federated Learning (FL) the process ends until the model converges or reached certain accuracy. My question: Is there any way to determine/estimate the number of rounds for the whole process?
2
votes
1answer
41 views

What is the most statistically acceptable method for tuning neural network hyperparameters on very small datasets?

Neural networks are usually evaluated by dividing a dataset into three splits: training, validation, and test The idea is that critical hyperparameters of the network such as the number of epochs ...
2
votes
1answer
50 views

Is it a good idea to use different width and height of the kernel in a CNN?

I always see that the width and height of the kernel are the same. But is it a good idea to use different numbers? Recently I tried to use GoogLeNet (which expects images to be 224x224) on my images (...
2
votes
0answers
24 views

How will the filter size affect the transpose convolution operation?

After a series of convolutions, I am up-sampling a compressed representation, I was curious what is the methodology I should follow to choose an optimum kernel size for up-sampling. How will the ...
1
vote
0answers
16 views

How does noise input size affect fake image generation with GANs?

In Generative Adversarial Networks, the Generator takes noise vector as input and feeds it forward to create an image. The noise vector consists of random numbers sampled from the normal distribution. ...
1
vote
1answer
38 views

For episodic tasks with an absorbing state, why can't we both have $\gamma=1$ and $T= \infty$ in the definition of the return?

For episodic tasks with an absorbing state, why can't $\gamma=1$ and $T= \infty$? In Sutton and Barto's book, they say that, for episodic tasks with absorbing states that becomes an infinite sequence, ...
2
votes
1answer
69 views

What should the value of epsilon be in the Q-learning?

I am trying to understand Reinforcement Learning and already explored different Youtube videos, blog posts, and Wikipedia articles. What I don't understand is the impact of $\epsilon$. What value ...
0
votes
0answers
126 views

How to determine the number of fully connected layers for a convolutional neural network?

How many fully connected layers should be added to a convolutional neural network? Does it depend on input size to the fully connected layer? If so, how do we decide? What if the input size of the ...
1
vote
2answers
49 views

Why is an embedding of dimension 400 enough to represent 70000 words?

I am learning PyTorch on Udacity. In lesson 8, section 11: Training the Model, the instructor writes: Then I have my embedding and hidden dimension. The embedding dimension is just a smaller ...
0
votes
0answers
44 views

How can I determine the k-NN's hyper-parameters and their range that significantly influence the outcome of classification?

Algorithm: Classification by k-nearest neighbors with Euclidean distance (neighbors.KNeighborsClassifier). Determine the important hyperparameters (2 maximum) that can significantly influence ...
1
vote
1answer
48 views

Should the range and initial values of weights and biases be adjusted to fit input and output data?

As a routine (in typical everyday tasks) of a data scientist, should they usually decide about weights and biases range and initial values as a function of which data they are planning to insert as ...
2
votes
1answer
94 views

Which hyperparameters in neural network are accesible to users adjustment

I am new to Neural Networks and my questions are still very basic. I know that most of neural networks allow and even ask user to chose hyper-parameters like: amount of hidden layers amount of ...
0
votes
0answers
35 views

How to optimize my GAN generator and discriminator models' structures?

I'm using Tensorflow to feed a DCGAN 3000 320x320 colored images of cars. The goal is to generate new cars. I've been training on Google Colab for the past 10 hours or so. I guess I can expect results ...
1
vote
1answer
35 views

Why one unit in the layers of neural network is not enough?

In a deep connected network, when every unit gets all the input features(X) so it has one parameter for every feature and every unit tweaks its parameters for loss optimization. What if we use only ...
2
votes
1answer
75 views

What is the intuition behind the number of filters/channels for each convolutional layer?

After having chosen the number of layers for a convolutional neural network, we must also choose the number of filters/channels for each convolutional layer. The intuition behind the filter's spatial ...
1
vote
1answer
225 views

What is the best activation function for the embedding layer in a deep auto-encoder?

I am designing a deep autoencoder for graph embedding (exactly node embedding) following this paper SDNE. In the original paper, they used the sigmoid activation for all hidden layers in the ...
1
vote
1answer
94 views

How to determine the number of hidden layers and units of a deep auto-encoder?

I am using a deep autoencoder for my problem. However, the way I choose the number of hidden layers and hidden units in a hidden layer is still based on my feeling. The size of the model that ...
3
votes
1answer
372 views

How should I choose the target's update frequency in DQN?

I have been dealing with a problem that I'm trying to solve with DQN. A general question that I have is regarding the target's update frequency. How should it change? Depending on what factor do we ...
3
votes
1answer
63 views

How are training hyperparameters determined for large models?

When training a relatively small DL model, which takes several hours to train, I typically start with some starting points from literature and then use a trial-and-error or grid-search approach to ...
3
votes
1answer
219 views

Why does every neuron in hidden layers of a multi-layer perceptron typically have the same activation function? [duplicate]

Why does every neuron in a hidden layer of a multi-layer perceptron (MLP) typically have the same activation function as every other neuron in the same or other hidden layers (so I exclude the output ...
2
votes
1answer
43 views

How exactly does nested cross-validation work?

I have trouble understanding how nested cross-validation works - I understand the need for two loops (one for selecting the model, and another for training the selected model), but why are they nested?...
2
votes
1answer
230 views

What made your DDPG implementation on your environment work?

I am working on scheduling problem that has inherent randomness. The dimensions of action and state spaces are 1 and 5 respectively. I am using DDPG, but it seems extremely unstable, and so far it ...
1
vote
0answers
34 views

Should we start with a small batch-size and increase during training to improve sample efficiency?

Just made an interesting observation playing around with the stable-baseline's implementation of PPO and the BipedalWalker environment from OpenAI's Gym. But I believe this should be a general ...
1
vote
1answer
40 views

How to know if the hyperparameters of a neural network relate to each other?

According this thread some hyperparameters are independent from each other while some are directly related. One of the answers give an example where two hyperparameters affect each other. For ...
1
vote
0answers
46 views

Why is the number of neurons used in various neural networks power of 2?

I have noticed that almost all tutorials take the number of neurons as a power of 2. Is there any proper mathematical and well-proven reason for that? If you sometimes change it to some other odd ...
1
vote
1answer
52 views

What is relation between gradient descent and regularization in deep learning?

Gradient descent is used to reduce the loss and regularization is used to fight over-fitting. Is there any relation between gradient descent and regularization, or both are independent of each other?...
3
votes
0answers
52 views

How can I do hyperparameter optimization for a CNN-LSTM neural network?

I have built a CNN-LSTM neural network with 2 inputs and 2 outputs in Keras. I trained the network with model.fit_generator() (and not ...
2
votes
0answers
30 views

Which CNN hyper-parameters are most sensitive to centered versus off centered data?

Which hyper-parameters of a convolutional neural network are likely to be the most sensitive to depending on whether the training (and test and inference) data involves only accurately centered images ...
3
votes
0answers
21 views

How to organize model training hyperparameters

I am working on multiple deep learning projects, most of them in the area of computer vision. For many of them I create multiple models, try different approaches, use various model architectures. And ...
6
votes
1answer
1k views

Is this idea to calculate the required number of hidden neurons for a single hidden layer neural network correct?

I have an idea to find the optimal number of hidden neurons required in a neural network, but I'm not sure how accurate it is. Assuming that it has only 1 hidden layer, it is a classification problem ...
2
votes
1answer
64 views

Will a .h5 file trained with Xception model work with Resnet50?

I have been running my 2013 server box since 2 weeks ago for training an AI model. I set up 30 epochs to run but since than it only ran 1 epoch as my PC config is super slow. But it generates 1 .h5 ...
2
votes
2answers
58 views

Does this hyperparameter optimisation approach yield the optimal hyperparameters?

Say I have a ML model which is not very costly to train. It has around say 5 hyperparameters. One way to select best hyperparameters would be to keep all the other hyperparamaters fixed and train ...
1
vote
0answers
302 views

Why is the $\epsilon$ hyper-parameter (in the $\epsilon$-greedy policy) annealed smoothly?

As far as I understand, RL is a process that can be divided into 2 stages: Exploring a wide range of paths (acting randomly) Refining the current optimal paths (revolving around actions with a so-...
2
votes
1answer
83 views

Can we automate the choice of the hyper-parameters of the evolutionary algorithms?

Certain hyper-parameters (e.g. the size of the offspring generation or the definition of the fitness function) and the design (e.g. how the mutation is performed) of evolutionary algorithms usually ...
2
votes
0answers
33 views

What is the benefit of scaling the hyperparameter C of an SVM?

Please read the following page of the Sklearn documentation. The figure shown there (see below) illustrates why C should be scaled when using a SVM with 'l1' penalty, whereas it shouldn't be scaled ...
9
votes
1answer
238 views

What causes a model to require a low learning rate?

I've pondered this for a while without developing an intuition for the math behind the cause of this. So what causes a model to need a low learning rate?
3
votes
1answer
53 views

How do you efficiently choose the hyper-parameters of a neural network?

How do you efficiently choose the hyper-parameters of a neural network (e.g. the learning rate, number of layer, weights, etc.)?
7
votes
1answer
581 views

How should we choose the dimensions of the encoding layer in auto-encoders?

How should we choose the dimensions of the encoding layer in auto-encoders?
2
votes
1answer
288 views

Maximum number of neurons in a layer given number of neurons in previous layer

Consider an extremely complicated feed-forward neural network training example but with no need of computational efficiency or limiting of processing time. What is the maximum number of hidden ...
0
votes
1answer
53 views

How many trees should be generated in a random forest?

What are ways of determining the number of trees to be generated in a random forest algorithm?
2
votes
2answers
203 views

What is the purpose of the “gamma” parameter in SVMs?

I want to understand what the gamma parameter does in an SVM. According to this page. Intuitively, the gamma parameter defines ...
4
votes
1answer
532 views

How does L2 regularization make weights smaller?

I'm learning logistic regression and $L_2$ regularization. The cost function looks like below. $$J(w) = -\displaystyle\sum_{i=1}^{n} (y^{(i)}\log(\phi(z^{(i)})+(1-y^{(i)})\log(1-\phi(z^{(i)})))$$ And ...
2
votes
1answer
101 views

Is a calculus or ML approach to varying learning rate as a function of loss and epoch been investigated?

Many have examined the idea of modifying learning rate at discrete times during the training of an artificial network using conventional back propagation. The goals of such work have been a balance ...
3
votes
1answer
65 views

What is the pros and cons of increasing and decreasing the number of worker processes in A3C?

In A3C, there are several child processes and one master process. The child precesses calculate the loss and backpropagation, and the master process sums them up and updates the parameters, if I ...
20
votes
3answers
9k views

How to choose an activation function for the hidden layers?

I choose the activation function for the output layer depending on the output that I need and the properties of the activation function that I know. For example, I choose the sigmoid function when I'm ...
7
votes
2answers
3k views

Why should the number of neurons in a hidden layer be a power of 2?

I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster. Is ...
8
votes
2answers
6k views

How do we choose the kernel size depending on the problem?

Obviously, finding suitable hyper-parameters for a neural network is a complex task and problem or domain-specific. However, there should be at least some "rules" that hold most times for the size of ...
3
votes
1answer
1k views

Why can't my implementation of the Actor-Critic algorithm achieve good results in the 2048 game?

I implemented the Actor-Critic with n-step TD prediction to learn to play the 2048 game For the environment, I don't use this 2048 implementation. I use a simple one without any graphical interface, ...
32
votes
4answers
1k views

How to find the optimal number of neurons per layer?

When you're writing your algorithm, how do you know how many neurons you need per single layer? Are there any methods for finding the optimal number of them, or is it a rule of thumb?