Questions tagged [hyperparameter-optimization]
For questions related to the concept of hyper-parameter optimization, that is, the task of finding the best hyper-parameters for a particular learning algorithm (e.g. gradient descent) or model (e.g. a multi-layer neural network) using an optimization method (e.g. Bayesian optimization or genetic algorithms).
125 questions
3
votes
1
answer
181
views
Can a ML model learn the hyperparameters landscape?
(I assume that this is not possible because I've never seen anyone talk about this.)
Let's take a classic MLP (named f) that, for example classify some images (from ...
1
vote
1
answer
39
views
Options for fitting a growth curve - process-based, hybrid, or neural networks?
I am trying to fit a Chapman-Richards growth curve:
$$
B = A*(1-e^{-kt})
$$
Where B is the biomass of a forest, A is the asymptote, k is the growth rate, and t is forest age. I expect the growth rate ...
0
votes
0
answers
12
views
Tuning loss weight in knowledge distillation
I am implementing a knowledge distillation model. However, the balance and the ratio between different loss components affect the knowledge distillation so much. Are there any good practice to find ...
0
votes
1
answer
45
views
When to stop a DQN agent train?
Hello AI Stack people,
I'm in doubt to when i should stop my DQN agent train.
I'm traning a DQN agent and i will use a hyperparameter optimization method (probabily random search). So, i need a ...
0
votes
1
answer
49
views
Why are biased results from ML models bad?
I am new to ML and am currently tinkering with scikit. At the moment I am using a random state both for the train_test split and a random state for my MLPClassifier, to tune hyperparameters with ...
- 3
2
votes
0
answers
82
views
VAE suffers from posterior collapse under all hyper parameters [closed]
I am trying to find a low-dimensional latent space representation for a bunch of simulated data. No matter what VAE architecture I try and no matter how I tweak it, the output of the VAE is always the ...
- 131
0
votes
0
answers
20
views
Combination of components to maximize a multi-criteria objective function
I have been given a list of components, with various “contributions” (or weights) which put together in a weighted combination have a combined aggregate effect. I then have the task of suggesting ...
- 101
0
votes
0
answers
24
views
Why can't I replicate the validation loss from a Keras tuner (LSTM)
I feel like I'm doing a pretty straightforward sequence of tasks and must be making a simple mistake - I simply build a sequential model, tune it, build a clone of the optimal model by extracting the ...
- 101
0
votes
0
answers
25
views
Imbue CARBS warning hyperparameter tuning
For a few iteration I got no warning but then it pops out telling me that
No Candidates found, choosing at random"
Here is the warning :
...
- 1
0
votes
0
answers
20
views
Optuna Hyperband Algorithm Not Following Expected Model Training Scheme
I have observed an issue while using the Hyperband algorithm in Optuna. According to the Hyperband algorithm, when min_resources = 5, max_resources = 20, and reduction_factor = 2, the search should ...
1
vote
0
answers
39
views
A Quest for Maximizing Expected Value: Optimizing Algorithmic Trading Parameters
I am engaged in algorithmic trading, employing specialized models that utilize various parameters to signal trading opportunities. Once I get a signal, I could execute a trade and everything is ...
- 11
0
votes
0
answers
31
views
Longer DNN training times when using evolutionary algorithms
I am comparing my deep neural network (DNN) performance when using 2 types of optimizers: gradient-based Adam (properly tuned) and a population-based optimization algorithm (e.g., genetic algorithm (...
1
vote
0
answers
110
views
Why would increasing layers in PyTorch Transformer significantly increase loss?
I have a simple torch.nn.Transformer module for machine translation on the Multi30k dataset. It performs pretty well (32.2 Bleu score) but I looked at scaling up ...
- 111
2
votes
1
answer
277
views
What is the basic difference between NAS, Hyperparameter Optimization/Tuning and Pruning?
As far as I understand:
NAS = an algorithm that searches for the best NN architecture, i.e., how many layers, what activation function to use, how many neurons, etc.
Hyperparameter Optim = finding ...
- 341
0
votes
1
answer
93
views
L2 regularization in BN layers, how to set gamma?
I have read tensorflow's documents about batch normzalization , but still don't get what is the gamma regulizer?
the link to document:
https://www.tensorflow.org/api_docs/python/tf/keras/layers/...
1
vote
1
answer
130
views
Are there any guidelines on picking hyperparameters for Deep Reinforcement Learning?
I am trying to learn machine learning from Andrew NG's Machine learning specialization on Coursera. In the chapter about reinforcement learning Andrew NG said that if you do not select correct ...
1
vote
1
answer
86
views
Are there techniques for estimating optimal neural network size?
Are there techniques for estimating optimal neural network size?
To replicate "AND gate", one does not need 1e1000 nodes in hidden layer. What would be the metric hinting at "too much ...
- 113
1
vote
2
answers
1k
views
What should I do if my validation score is good, but my test score is bad?
I've trained my artificial neural network, and, as per standard practice, I've picked out the one neural network throughout training that did the best on my validation dataset. That is, the neural ...
- 113
-1
votes
0
answers
23
views
How to speed up the learning process
I have built a network that performans pretty well on my data. The issue I have is that for a larger number of epochs at the start of the training process the val/train acc/loss are stagnating (for ...
- 57
1
vote
0
answers
48
views
How to make my neural networks designs more robust
Whenever, I design a neural network to solve a novel problem (requires a novel loss function i.e. not image classification) it always ends up being hypersensitive to batch size and learning rate.
...
- 366
0
votes
1
answer
903
views
Why does GridSearchCV model give worse results despite same parameters used with base model
I am trying to make prediction using random forest regression and then utilize GridSearchCV to tune hyperparameters(just 'n_estimators'). However results of GridSearchCV are worse than base model. ...
1
vote
1
answer
126
views
Hyperparameter tuning methods for neural networks
I have a fully connected feedforward classifier neural network that uses the leaky ReLU activation function. I would like to apply a state-of-the-art hyperparameter tuning method to my methodology. ...
- 111
0
votes
1
answer
82
views
Cross Validation and hyperparameter selection correct procedure
I am trying to run a regression supervised learning problem. The dataset is not very large and I wanted to do some cross-validation to avoid overfitting. As I have read it's important to do a ...
- 1
1
vote
2
answers
73
views
How to manually adjust output from model? [closed]
I wonder if it is possible to add manual inference to the output of a model?
For example, I have a model called 'net', and the output value of 'net' is a vector called v = [v1, ... vn]. v is a binary ...
- 111
3
votes
0
answers
45
views
How should I compare multiple machine learning models to be (generally) fair to all models?
I am testing multiple models on IBM HR Analytics Attrition Dataset (1470 lines) and HR Analytics dataset (15000 lines) for a research project. The models include traditional models (Naive Bayes, SVM), ...
- 131
2
votes
3
answers
243
views
Is it possible to learn the number of layers?
Is it possible, in a transformer or other deep architecture, to include the number of layers as a parameter of the model so it could be learned?
In fact, I have a keras layer that I use to change the ...
- 51
0
votes
0
answers
28
views
Resolving Derivation Discrepancies for Differentiating through Optimization Paths
I'm reading the paper "Optimizing Millions of Hyperparameters by Implicit Differentiation". The key contribution of the paper is to show that you can replace optimizing through the ...
- 1
3
votes
1
answer
1k
views
How do I design the network for Deep Q-Network?
I am playing with deep q-learning and I am thinking about what the proper architecture of a network used for deep q-learning is.
I have a very simple environment, basically a 18x18 matrix, where 3 ...
- 133
1
vote
1
answer
288
views
How can I interpret the value returned by score(X) method of sklearn.neighbors.KernelDensity?
For sklearn.neighbors.KernelDensity, its score(X) method according to the sklearn KDE documentation says:
Compute the log-...
- 235
1
vote
1
answer
679
views
How does stochastic gradient descent undo the normalization done by the batch normalization?
I want to understand the handshake between SGD (or mini-batch GD) and batch normalization. Below, an explanation quoted from this Medium article. However, I am confused about the denormalization by ...
- 13
1
vote
2
answers
1k
views
Random forests - are more estimators always better?
I'm learning about more advanced methods of hyperparameter optimization, such as the Bayesian methods in the scikit-optimize package. For those unfamiliar with the ...
- 271
2
votes
1
answer
111
views
What components of reinforcement learning influence the result the most?
I'm working on my thesis concerning a reinforcement learning problem and am trying to prioritise my time on different components of it:
Formalising the agent environment (like the design of state-, ...
- 119
0
votes
0
answers
38
views
Is there a way to adapt Particle Swarm Optimization to an incremental/online learning setting?
As stated in the title, is there a way to adapt PSO to an online scenario where new data samples arrive continuously?
In more detail: suppose that I have a classifier with several parameters for which ...
- 1
0
votes
2
answers
5k
views
How to fine-tune GPT-J with small dataset
I have followed this guide as closely as possible: https://github.com/kingoflolz/mesh-transformer-jax
I'm trying to fine-tune GPT-J with a small dataset of ~500 lines:
...
- 101
5
votes
2
answers
773
views
Do we need automatic hyper-parameter tuning when we have a large enough dataset?
Hyperparameter tuning is the process of selecting the optimal hyperparameters for an ANN.
Now, my guess is that, if we have sufficient data (say, 1.4 million for, say, 6 features), the model can be ...
- 341
3
votes
1
answer
348
views
How do I choose the hyper-parameters for a model to detect different guitar chords?
I need to build a hand detector that recognizes the chord played by a hand on a guitar.
I read this article Static Hand Gesture Recognition using Convolutional Neural Network with Data Augmentation ...
- 83
1
vote
0
answers
36
views
How do you decide that you have tested enough hyper-parameter combinations for a specific neural network architecture?
How do you decide that you have tested enough hyper-parameter combinations for a specific neural network architecture to discard it and move on to a new model?
Do you have a structured (generic) ...
- 11
2
votes
0
answers
129
views
Has the idea of using different learning rates for different layers been explored in the literature?
I wonder whether there are heuristic rules for the optimal selection of learning rates for different layers. I expect that there is no general recipe, but probably there are some choices that may be ...
- 2,754
1
vote
1
answer
148
views
Can the optimal learning rate differ for different architectures?
In several courses and tutorials about neural networks, people often say that the learning rate (LR) should be the first hyper-parameter to be tuned before we tweak the others. For example, in this ...
- 1,280
1
vote
0
answers
72
views
Why doesn't anyone use reinforcement learning to find the best possible alternative to backpropagation?
To be clear, I'm very uninformed on the topic of alternative learning algorithms to backprop, all my knowledge comes from articles like these:
lets-not-stop-at-backprop
backprop-alternatives
we-need-a-...
- 103
13
votes
3
answers
24k
views
How to determine the embedding size?
When we are training a neural network, we are going to determine the embedding size to convert the categorical (in NLP, for instance) or continuous (in computer vision or voice) information to hidden ...
- 967
0
votes
1
answer
1k
views
Is it valid to implement hyper-parameter tuning and THEN cross-validation?
I have a multi-label classification task I am solving. I have done hyperparameter tuning (with Keras Tuner) to determine the best configuration for my neural network.
Is it valid to do this (determine ...
- 171
1
vote
1
answer
212
views
How many singular vectors do we need to calculate for SVD?
In the geometrical interpretation of SVD, the data points that we have need to be imagined as points in high dimensional space (say $d$-dimensional space). But we need to find a hyperplane in $k-$...
- 3,990
1
vote
0
answers
39
views
Are there guiding principles as to which activation functions suit a given RL algorithm?
Are there rules of thumb as to which activation functions work well (or which one would not) on the policy and value network of a class of RL algorithms? For hidden layers and for the output layer.
...
- 541
7
votes
2
answers
4k
views
What are the best hyper-parameters to tune in reinforcement learning?
Obviously, this is somewhat subjective, but what hyper-parameters typically have the most significant impact on an RL agent's ability to learn? For example, the replay buffer size, learning rate, ...
- 293
2
votes
0
answers
56
views
Is there an optimal number of species for NEAT?
Is there an optimal number of species for NEAT?
Since too low and too high is bad, I am thinking about adjusting the threshold of the distance function at runtime in order to have the number of ...
- 51
1
vote
0
answers
421
views
Why does Adam optimizer work slower than Adagrad, Adadelta, and SGD for Neural Collaborative Filtering (NCF)?
I've been working on Neural Collaborative Filtering (NCF) recently to build a recommender system using Tensorflow Recommenders. Doing some hyperparameter tuning with different optimizers available in ...
- 111
3
votes
1
answer
116
views
What is the most statistically acceptable method for tuning neural network hyperparameters on very small datasets?
Neural networks are usually evaluated by dividing a dataset into three splits:
training,
validation, and
test
The idea is that critical hyperparameters of the network such as the number of epochs ...
- 411
0
votes
2
answers
1k
views
Do larger numbers of hidden layers have a bigger effect on a classification model's accuracy?
I trained different classification models using Keras with different numbers of hidden layers and the same number of neurons in each layer. What I found was the accuracy of the models decreased as the ...
3
votes
0
answers
594
views
How to Select Model Parameters for Transformer (Heads, number of layers, etc)
Is there a general guideline on how the Transformer model parameters should be selected, or the range of these parameters that should be included in a hyperparameter sweep?
Number of heads
Number of ...
- 351