Questions tagged [hyperparameter-optimization]
For questions related to the concept of hyper-parameter optimization, that is, the task of finding the best hyper-parameters for a particular learning algorithm (e.g. gradient descent) or model (e.g. a multi-layer neural network) using an optimization method (e.g. Bayesian optimization or genetic algorithms).
122
questions
1
vote
0
answers
29
views
A Quest for Maximizing Expected Value: Optimizing Algorithmic Trading Parameters
I am engaged in algorithmic trading, employing specialized models that utilize various parameters to signal trading opportunities. Once I get a signal, I could execute a trade and everything is ...
- 11
0
votes
0
answers
9
views
Incorporating HiPlot and Keras
I just started to learn about Keras and train some models, and I came across HiPlot which is used for tuning hyperparameters. I was wondering if HiPlot can also be used to see what parameters would ...
- 1
0
votes
0
answers
23
views
Longer DNN training times when using evolutionary algorithms
I am comparing my deep neural network (DNN) performance when using 2 types of optimizers: gradient-based Adam (properly tuned) and a population-based optimization algorithm (e.g., genetic algorithm (...
1
vote
0
answers
41
views
Why would increasing layers in PyTorch Transformer significantly increase loss?
I have a simple torch.nn.Transformer module for machine translation on the Multi30k dataset. It performs pretty well (32.2 Bleu score) but I looked at scaling up ...
- 111
2
votes
1
answer
89
views
What is the basic difference between NAS, Hyperparameter Optimization/Tuning and Pruning?
As far as I understand:
NAS = an algorithm that searches for the best NN architecture, i.e., how many layers, what activation function to use, how many neurons, etc.
Hyperparameter Optim = finding ...
- 351
0
votes
1
answer
48
views
L2 regularization in BN layers, how to set gamma?
I have read tensorflow's documents about batch normzalization , but still don't get what is the gamma regulizer?
the link to document:
https://www.tensorflow.org/api_docs/python/tf/keras/layers/...
0
votes
0
answers
9
views
Constrain or not constrain overlapping architectural choices when wanting to compare deep neural architectures
I want to compare various deep (recurrent) neural architectures and was wondering what the best approach is. The models in question all use several LSTMs/GRUs etc.
Fine-tune all models fully and ...
- 1,898
1
vote
1
answer
83
views
Are there any guidelines on picking hyperparameters for Deep Reinforcement Learning?
I am trying to learn machine learning from Andrew NG's Machine learning specialization on Coursera. In the chapter about reinforcement learning Andrew NG said that if you do not select correct ...
0
votes
0
answers
28
views
Are there techniques (pygad - tpot-optuna)for best Genatic algorithm optimize hyper prameter cnn 1D
Iam new in mashin learning and i try to optimize tenser flow with keras conv1d model to improve classification by improve hyper kernal and filter
For training dataset csv =(1325,33,1) with outputs ...
- 1
1
vote
1
answer
75
views
Are there techniques for estimating optimal neural network size?
Are there techniques for estimating optimal neural network size?
To replicate "AND gate", one does not need 1e1000 nodes in hidden layer. What would be the metric hinting at "too much ...
- 113
0
votes
2
answers
298
views
What should I do if my validation score is good, but my test score is bad?
I've trained my artificial neural network, and, as per standard practice, I've picked out the one neural network throughout training that did the best on my validation dataset. That is, the neural ...
- 103
0
votes
0
answers
51
views
Why the accuracy of a set of hypermodels predicting a sequence never goes beyond a small number
I have been trying to use Optuna to optimize a model predicting an integer sequence taking 5 numbers to predict the next, but the validation accuracy never gets beyond a little more than 13% no matter ...
0
votes
0
answers
19
views
How to speed up the learning process
I have built a network that performans pretty well on my data. The issue I have is that for a larger number of epochs at the start of the training process the val/train acc/loss are stagnating (for ...
- 51
0
votes
0
answers
111
views
Optimal weight decay value in Adam
Is there any rule of thumb while assigning the weight_decay parameter in Adam optimizer?
As in, is it somehow related to (smaller or larger than) the learning rate ...
- 65
1
vote
0
answers
23
views
How to make my neural networks designs more robust
Whenever, I design a neural network to solve a novel problem (requires a novel loss function i.e. not image classification) it always ends up being hypersensitive to batch size and learning rate.
...
- 316
0
votes
0
answers
130
views
VAE - Which loss to optimize for?
Regarding hyperparameter optimization for VAEs. Should you optimize for the reconstruction loss, or the complete ELBO (- KL divergence + reconstruction loss)?
My thought is that it probably depends on ...
- 11
0
votes
1
answer
349
views
Why does GridSearchCV model give worse results despite same parameters used with base model
I am trying to make prediction using random forest regression and then utilize GridSearchCV to tune hyperparameters(just 'n_estimators'). However results of GridSearchCV are worse than base model. ...
0
votes
0
answers
86
views
cannot get the results for each individual execution in hyperband tuner
I am trying to use keras tuner hyperband to select hyperparameters for my autoencoder model. Here's some pseudo code:
...
1
vote
1
answer
98
views
Hyperparameter tuning methods for neural networks
I have a fully connected feedforward classifier neural network that uses the leaky ReLU activation function. I would like to apply a state-of-the-art hyperparameter tuning method to my methodology. ...
- 111
0
votes
1
answer
66
views
Cross Validation and hyperparameter selection correct procedure
I am trying to run a regression supervised learning problem. The dataset is not very large and I wanted to do some cross-validation to avoid overfitting. As I have read it's important to do a ...
- 1
1
vote
2
answers
48
views
How to manually adjust output from model? [closed]
I wonder if it is possible to add manual inference to the output of a model?
For example, I have a model called 'net', and the output value of 'net' is a vector called v = [v1, ... vn]. v is a binary ...
- 111
3
votes
0
answers
30
views
How should I compare multiple machine learning models to be (generally) fair to all models?
I am testing multiple models on IBM HR Analytics Attrition Dataset (1470 lines) and HR Analytics dataset (15000 lines) for a research project. The models include traditional models (Naive Bayes, SVM), ...
- 131
2
votes
3
answers
221
views
Is it possible to learn the number of layers?
Is it possible, in a transformer or other deep architecture, to include the number of layers as a parameter of the model so it could be learned?
In fact, I have a keras layer that I use to change the ...
- 51
0
votes
0
answers
27
views
Resolving Derivation Discrepancies for Differentiating through Optimization Paths
I'm reading the paper "Optimizing Millions of Hyperparameters by Implicit Differentiation". The key contribution of the paper is to show that you can replace optimizing through the ...
- 1
3
votes
1
answer
322
views
How do I design the network for Deep Q-Network?
I am playing with deep q-learning and I am thinking about what the proper architecture of a network used for deep q-learning is.
I have a very simple environment, basically a 18x18 matrix, where 3 ...
- 133
1
vote
1
answer
169
views
How can I interpret the value returned by score(X) method of sklearn.neighbors.KernelDensity?
For sklearn.neighbors.KernelDensity, its score(X) method according to the sklearn KDE documentation says:
Compute the log-...
- 225
1
vote
1
answer
444
views
How does stochastic gradient descent undo the normalization done by the batch normalization?
I want to understand the handshake between SGD (or mini-batch GD) and batch normalization. Below, an explanation quoted from this Medium article. However, I am confused about the denormalization by ...
- 13
1
vote
2
answers
606
views
Random forests - are more estimators always better?
I'm learning about more advanced methods of hyperparameter optimization, such as the Bayesian methods in the scikit-optimize package. For those unfamiliar with the ...
- 273
2
votes
1
answer
95
views
What components of reinforcement learning influence the result the most?
I'm working on my thesis concerning a reinforcement learning problem and am trying to prioritise my time on different components of it:
Formalising the agent environment (like the design of state-, ...
- 115
0
votes
0
answers
30
views
Is there a way to adapt Particle Swarm Optimization to an incremental/online learning setting?
As stated in the title, is there a way to adapt PSO to an online scenario where new data samples arrive continuously?
In more detail: suppose that I have a classifier with several parameters for which ...
- 1
0
votes
2
answers
5k
views
How to fine-tune GPT-J with small dataset
I have followed this guide as closely as possible: https://github.com/kingoflolz/mesh-transformer-jax
I'm trying to fine-tune GPT-J with a small dataset of ~500 lines:
...
- 101
5
votes
2
answers
755
views
Do we need automatic hyper-parameter tuning when we have a large enough dataset?
Hyperparameter tuning is the process of selecting the optimal hyperparameters for an ANN.
Now, my guess is that, if we have sufficient data (say, 1.4 million for, say, 6 features), the model can be ...
- 351
3
votes
1
answer
243
views
How do I choose the hyper-parameters for a model to detect different guitar chords?
I need to build a hand detector that recognizes the chord played by a hand on a guitar.
I read this article Static Hand Gesture Recognition using Convolutional Neural Network with Data Augmentation ...
- 83
1
vote
0
answers
34
views
How do you decide that you have tested enough hyper-parameter combinations for a specific neural network architecture?
How do you decide that you have tested enough hyper-parameter combinations for a specific neural network architecture to discard it and move on to a new model?
Do you have a structured (generic) ...
- 11
2
votes
0
answers
71
views
Has the idea of using different learning rates for different layers been explored in the literature?
I wonder whether there are heuristic rules for the optimal selection of learning rates for different layers. I expect that there is no general recipe, but probably there are some choices that may be ...
- 2,494
1
vote
1
answer
107
views
Can the optimal learning rate differ for different architectures?
In several courses and tutorials about neural networks, people often say that the learning rate (LR) should be the first hyper-parameter to be tuned before we tweak the others. For example, in this ...
- 1,240
1
vote
0
answers
65
views
Why doesn't anyone use reinforcement learning to find the best possible alternative to backpropagation?
To be clear, I'm very uninformed on the topic of alternative learning algorithms to backprop, all my knowledge comes from articles like these:
lets-not-stop-at-backprop
backprop-alternatives
we-need-a-...
- 103
7
votes
3
answers
14k
views
How to determine the embedding size?
When we are training a neural network, we are going to determine the embedding size to convert the categorical (in NLP, for instance) or continuous (in computer vision or voice) information to hidden ...
- 877
0
votes
1
answer
710
views
Is it valid to implement hyper-parameter tuning and THEN cross-validation?
I have a multi-label classification task I am solving. I have done hyperparameter tuning (with Keras Tuner) to determine the best configuration for my neural network.
Is it valid to do this (determine ...
- 161
1
vote
1
answer
167
views
How many singular vectors do we need to calculate for SVD?
In the geometrical interpretation of SVD, the data points that we have need to be imagined as points in high dimensional space (say $d$-dimensional space). But we need to find a hyperplane in $k-$...
- 3,581
1
vote
0
answers
35
views
Are there guiding principles as to which activation functions suit a given RL algorithm?
Are there rules of thumb as to which activation functions work well (or which one would not) on the policy and value network of a class of RL algorithms? For hidden layers and for the output layer.
...
- 531
5
votes
2
answers
3k
views
What are the best hyper-parameters to tune in reinforcement learning?
Obviously, this is somewhat subjective, but what hyper-parameters typically have the most significant impact on an RL agent's ability to learn? For example, the replay buffer size, learning rate, ...
- 243
2
votes
0
answers
41
views
Is there an optimal number of species for NEAT?
Is there an optimal number of species for NEAT?
Since too low and too high is bad, I am thinking about adjusting the threshold of the distance function at runtime in order to have the number of ...
- 51
1
vote
0
answers
239
views
Why does Adam optimizer work slower than Adagrad, Adadelta, and SGD for Neural Collaborative Filtering (NCF)?
I've been working on Neural Collaborative Filtering (NCF) recently to build a recommender system using Tensorflow Recommenders. Doing some hyperparameter tuning with different optimizers available in ...
- 111
3
votes
1
answer
101
views
What is the most statistically acceptable method for tuning neural network hyperparameters on very small datasets?
Neural networks are usually evaluated by dividing a dataset into three splits:
training,
validation, and
test
The idea is that critical hyperparameters of the network such as the number of epochs ...
- 401
0
votes
2
answers
640
views
Do larger numbers of hidden layers have a bigger effect on a classification model's accuracy?
I trained different classification models using Keras with different numbers of hidden layers and the same number of neurons in each layer. What I found was the accuracy of the models decreased as the ...
3
votes
0
answers
442
views
How to Select Model Parameters for Transformer (Heads, number of layers, etc)
Is there a general guideline on how the Transformer model parameters should be selected, or the range of these parameters that should be included in a hyperparameter sweep?
Number of heads
Number of ...
- 321
0
votes
1
answer
692
views
What to do with a GAN that trained well but got worse over time?
I am training a WGAN-GP network based on the following paper, though I am using a different dataset. Now, for the first ~ 60-70 epochs, my network trained really well, which I could see in the loss ...
- 171
2
votes
1
answer
56
views
Is it possible to train one part of the network with a particular learning rate and the other part with a different one?
I have a combined network consisting of two parts: one is for images and the other is for numerical data. Each sample is matched with a numerical case by an ID. For this combined network, a ...
- 241
4
votes
1
answer
133
views
Bayesian hyperparameter optimization, is it worth it?
In the Deep Learning book by Goodfellow et al., section 11.4.5 (p. 438), the following claims can be found:
Currently, we cannot unambiguously recommend Bayesian hyperparameter
optimization as an ...
- 183