Questions tagged [hyper-parameters]

For questions related to the hyper-parameters of AI models and algorithms, which are parameters that are set before the learning process begins. For example, the number of hidden layers in a feed-forward neural network is usually a hyper-parameter.

Filter by
Sorted by
Tagged with
0
votes
1answer
25 views

What can I infer if my model is converging extremely fast?

I am running a model with fixed hyperparameters. To my surprise/shock, the model converged extremely fast with the least loss possible. I want to know the causes of this phenomenon. I have the ...
0
votes
2answers
87 views

Is it true that batch size of form $2^k$ gives better results?

I am confused among the following in selecting the batch size for my model. #1: powers of 2 I generally see that batch sizes are in powers of two: 32, 64, 128, 256. #2: maximum GPU Suppose my GPU ...
2
votes
1answer
68 views

What components of reinforcement learning influence the result the most?

I'm working on my thesis concerning a reinforcement learning problem and am trying to prioritise my time on different components of it: Formalising the agent environment (like the design of state-, ...
3
votes
2answers
420 views

When can I call an entity a hyperparameter?

As per my knowledge, any entity that is learnable by a training algorithm can be called a parameter. Weights of a neural network are called parameters because of this reason only. But I have doubts ...
0
votes
2answers
61 views

Is there any relationship between the batch size and the number of epochs?

I am currently running a program with a batch size of 17 instead of batch size 32. The benchmark results are obtained at a batch size of 32 with the number of epochs 700. Now I am running with batch ...
0
votes
0answers
18 views

How Long Can BPTT Truncated?

I wanted to ask what is, in general, the maximum value (the order of magnitude) of the number of time steps I can back-propagate in the past using TBTT (Truncated Backpropagation Through Time) in an ...
0
votes
1answer
23 views

How to reduce the number of clusters produced by the Markov Clustering Algorithm?

I have used the Markov Clustering Algorithm (MCL) to cluster tweets, based on their similarity. However, I got a too high number of clusters, and most of the clusters have only one tweet. Any ...
0
votes
0answers
9 views

Loss & accuracy curves from learning rate range test interpretation

I am working on a project doing experiments with the Learning Rate Range Test (See "A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and ...
0
votes
2answers
45 views

Why data required for hyperparameter tuning is considered as an additional data?

Any parametric model may have parameters as well as hyperparameters. Learning algorithm deals with parameters and hyperparameters should be dealt outside learning algorithm. Consider the following ...
1
vote
1answer
59 views

Why is the input layer of a neural network usually not counted?

I came across the following statement from the caption of figure 7.8 from the textbook Neural Networks and Neural Language Models the input layer is usually not counted when enumerating layers Why ...
2
votes
2answers
226 views

How to determine the embedding size?

When we are training a neural network, we are going to determine the embedding size to convert the categorical (in NLP, for instance) or continuous (in computer vision or voice) information to hidden ...
1
vote
1answer
47 views

How many singular vectors do we need to calculate for SVD?

In the geometrical interpretation of SVD, the data points that we have need to be imagined as points in high dimensional space (say $d$-dimensional space). But we need to find a hyperplane in $k-$...
0
votes
0answers
34 views

How to change number of trained layers in object detection TensorFlow models?

Training custom object detection models with TensorFlow usually means a transfer learning of pre-trained models and, if I understand it correctly, it means only training the few last layers, with ...
0
votes
2answers
46 views

How to ensure that the ES-HyperNEAT algorithm generates an ANN in the substrate?

I'm trying to implement the ES-HyperNEAT algorithm using the original paper, as well as the pseudocode provided in the official user page. Occasionally, the algorithm would be unable to generate a ...
0
votes
0answers
344 views

Optimal episode length in reinforcement learning

I have a custom environment for stock trading where an episode can be as long as 2000-3000 steps. I've run several experiments with td3 and sac algorithms, average reward per episode flattens after ...
3
votes
2answers
117 views

What are the best hyper-parameters to tune in reinforcement learning?

Obviously, this is somewhat subjective, but what hyper-parameters typically have the most significant impact on an RL agent's ability to learn? For example, the replay buffer size, learning rate, ...
2
votes
0answers
22 views

Is there an optimal number of species for NEAT?

Is there an optimal number of species for NEAT? Since too low and too high is bad, I am thinking about adjusting the threshold of the distance function at runtime in order to have the number of ...
1
vote
0answers
18 views

Hyperparameters for Reproducing the Results of IRGAN on MovieLens 1M

I am trying to reproduce results reported for IRGAN (information retrieval GAN) on the MovieLens 1M dataset. The results I want to reproduce and their sources are listed in the table below. Model ...
0
votes
0answers
42 views

Is there any way to determine/estimate the number of rounds for the whole Federated Learning process?

In Federated Learning (FL) the process ends until the model converges or reached certain accuracy. My question: Is there any way to determine/estimate the number of rounds for the whole process?
3
votes
1answer
60 views

What is the most statistically acceptable method for tuning neural network hyperparameters on very small datasets?

Neural networks are usually evaluated by dividing a dataset into three splits: training, validation, and test The idea is that critical hyperparameters of the network such as the number of epochs ...
2
votes
1answer
146 views

Is it a good idea to use different width and height of the kernel in a CNN?

I always see that the width and height of the kernel are the same. But is it a good idea to use different numbers? Recently I tried to use GoogLeNet (which expects images to be 224x224) on my images (...
2
votes
0answers
37 views

Given a 2-layer GCN, can we choose the dimensions of the 2nd weight matrix, such that this architecture has the same capacity as a 1-layer GCN?

This might be more of a question about nested function classes: For $k$ class node classification in a graph with $n$ nodes, and $d$ feature vector. I want to compare Architecture I: the GCN model of ...
2
votes
0answers
58 views

How will the filter size affect the transpose convolution operation?

After a series of convolutions, I am up-sampling a compressed representation, I was curious what is the methodology I should follow to choose an optimum kernel size for up-sampling. How will the ...
1
vote
0answers
31 views

How does noise input size affect fake image generation with GANs?

In Generative Adversarial Networks, the Generator takes noise vector as input and feeds it forward to create an image. The noise vector consists of random numbers sampled from the normal distribution. ...
1
vote
1answer
46 views

For episodic tasks with an absorbing state, why can't we both have $\gamma=1$ and $T= \infty$ in the definition of the return?

For episodic tasks with an absorbing state, why can't $\gamma=1$ and $T= \infty$? In Sutton and Barto's book, they say that, for episodic tasks with absorbing states that becomes an infinite sequence, ...
2
votes
1answer
263 views

What should the value of epsilon be in the Q-learning?

I am trying to understand Reinforcement Learning and already explored different Youtube videos, blog posts, and Wikipedia articles. What I don't understand is the impact of $\epsilon$. What value ...
1
vote
2answers
106 views

Why is an embedding of dimension 400 enough to represent 70000 words?

I am learning PyTorch on Udacity. In lesson 8, section 11: Training the Model, the instructor writes: Then I have my embedding and hidden dimension. The embedding dimension is just a smaller ...
1
vote
1answer
53 views

Should the range and initial values of weights and biases be adjusted to fit input and output data?

As a routine (in typical everyday tasks) of a data scientist, should they usually decide about weights and biases range and initial values as a function of which data they are planning to insert as ...
2
votes
1answer
102 views

Which hyperparameters in neural network are accesible to users adjustment

I am new to Neural Networks and my questions are still very basic. I know that most of neural networks allow and even ask user to chose hyper-parameters like: amount of hidden layers amount of ...
1
vote
1answer
36 views

Why one unit in the layers of neural network is not enough?

In a deep connected network, when every unit gets all the input features(X) so it has one parameter for every feature and every unit tweaks its parameters for loss optimization. What if we use only ...
2
votes
1answer
188 views

What is the intuition behind the number of filters/channels for each convolutional layer?

After having chosen the number of layers for a convolutional neural network, we must also choose the number of filters/channels for each convolutional layer. The intuition behind the filter's spatial ...
1
vote
1answer
626 views

What is the best activation function for the embedding layer in a deep auto-encoder?

I am designing a deep autoencoder for graph embedding (exactly node embedding) following this paper SDNE. In the original paper, they used the sigmoid activation for all hidden layers in the ...
1
vote
1answer
183 views

How to determine the number of hidden layers and units of a deep auto-encoder?

I am using a deep autoencoder for my problem. However, the way I choose the number of hidden layers and hidden units in a hidden layer is still based on my feeling. The size of the model that ...
3
votes
1answer
843 views

How should I choose the target's update frequency in DQN?

I have been dealing with a problem that I'm trying to solve with DQN. A general question that I have is regarding the target's update frequency. How should it change? Depending on what factor do we ...
3
votes
1answer
71 views

How are training hyperparameters determined for large models?

When training a relatively small DL model, which takes several hours to train, I typically start with some starting points from literature and then use a trial-and-error or grid-search approach to ...
3
votes
1answer
388 views

Why does every neuron in hidden layers of a multi-layer perceptron typically have the same activation function? [duplicate]

Why does every neuron in a hidden layer of a multi-layer perceptron (MLP) typically have the same activation function as every other neuron in the same or other hidden layers (so I exclude the output ...
2
votes
1answer
48 views

How exactly does nested cross-validation work?

I have trouble understanding how nested cross-validation works - I understand the need for two loops (one for selecting the model, and another for training the selected model), but why are they nested?...
3
votes
1answer
396 views

What made your DDPG implementation on your environment work?

I am working on scheduling problem that has inherent randomness. The dimensions of action and state spaces are 1 and 5 respectively. I am using DDPG, but it seems extremely unstable, and so far it ...
1
vote
0answers
36 views

Should we start with a small batch-size and increase during training to improve sample efficiency?

Just made an interesting observation playing around with the stable-baseline's implementation of PPO and the BipedalWalker environment from OpenAI's Gym. But I believe this should be a general ...
1
vote
1answer
76 views

How to know if the hyperparameters of a neural network relate to each other?

According this thread some hyperparameters are independent from each other while some are directly related. One of the answers give an example where two hyperparameters affect each other. For ...
1
vote
0answers
46 views

Why is the number of neurons used in various neural networks power of 2?

I have noticed that almost all tutorials take the number of neurons as a power of 2. Is there any proper mathematical and well-proven reason for that? If you sometimes change it to some other odd ...
1
vote
1answer
56 views

What is relation between gradient descent and regularization in deep learning?

Gradient descent is used to reduce the loss and regularization is used to fight over-fitting. Is there any relation between gradient descent and regularization, or both are independent of each other?...
3
votes
0answers
57 views

How can I do hyperparameter optimization for a CNN-LSTM neural network?

I have built a CNN-LSTM neural network with 2 inputs and 2 outputs in Keras. I trained the network with model.fit_generator() (and not ...
2
votes
0answers
43 views

Which CNN hyper-parameters are most sensitive to centered versus off centered data?

Which hyper-parameters of a convolutional neural network are likely to be the most sensitive to depending on whether the training (and test and inference) data involves only accurately centered images ...
3
votes
0answers
22 views

How to organize model training hyperparameters

I am working on multiple deep learning projects, most of them in the area of computer vision. For many of them I create multiple models, try different approaches, use various model architectures. And ...
3
votes
0answers
31 views

Which hyper-parameters are considered in neural architecture search?

I want to understand automatic Neural Architecture Search (NAS). I read already multiple papers, but I cannot figure out what the actual search space of NAS is / how are classical hyper-parameters ...
4
votes
1answer
55 views

When using Neural Architecture Search, how are the hyper-parameters chosen?

I have read a lot about NAS, but I still do not understand one concept: When setting up a neural network, hyperparameters (such as the learning rate, dropout rate, batch size, filter size, etc.) need ...
4
votes
1answer
695 views

How many weights does the max-pooling layer have?

How many weights does the max-pooling layer have? For example, if there are 10 inputs, a pooling filter of size 2, stride 2, how many weights, including bias, does a max-pooling layer have?
6
votes
1answer
1k views

Is this idea to calculate the required number of hidden neurons for a single hidden layer neural network correct?

I have an idea to find the optimal number of hidden neurons required in a neural network, but I'm not sure how accurate it is. Assuming that it has only 1 hidden layer, it is a classification problem ...
1
vote
1answer
80 views

Will a .h5 file trained with Xception model work with Resnet50? [closed]

I have been running my 2013 server box since 2 weeks ago for training an AI model. I set up 30 epochs to run but since than it only ran 1 epoch as my PC config is super slow. But it generates 1 .h5 ...