Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [hyperparameter-optimization]

For questions related to the concept of hyper-parameter optimization, that is, the task of finding the best hyper-parameters for a particular learning algorithm (e.g. gradient descent) or model (e.g. a multi-layer neural network) using an optimization method (e.g. Bayesian optimization or genetic algorithms).

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

Optuna Hyperband Algorithm Not Following Expected Model Training Scheme

I have observed an issue while using the Hyperband algorithm in Optuna. According to the Hyperband algorithm, when min_resources = 5, max_resources = 20, and reduction_factor = 2, the search should ...
Tnb Marketplace's user avatar
0 votes
0 answers
17 views

Output Not Changing - Feeding Previous Outputs Back Into a Model

Full disclosure, I also posted this on Stack Overflow I have put a more theory based bent towards the question itself here I have a simple model in pytorch based on the quickstart except instead of a ...
cgbsu's user avatar
  • 1
1 vote
0 answers
36 views

A Quest for Maximizing Expected Value: Optimizing Algorithmic Trading Parameters

I am engaged in algorithmic trading, employing specialized models that utilize various parameters to signal trading opportunities. Once I get a signal, I could execute a trade and everything is ...
David's user avatar
  • 11
0 votes
0 answers
12 views

Incorporating HiPlot and Keras

I just started to learn about Keras and train some models, and I came across HiPlot which is used for tuning hyperparameters. I was wondering if HiPlot can also be used to see what parameters would ...
Kasiopea's user avatar
0 votes
0 answers
25 views

Longer DNN training times when using evolutionary algorithms

I am comparing my deep neural network (DNN) performance when using 2 types of optimizers: gradient-based Adam (properly tuned) and a population-based optimization algorithm (e.g., genetic algorithm (...
knowledge_seeker's user avatar
1 vote
0 answers
87 views

Why would increasing layers in PyTorch Transformer significantly increase loss?

I have a simple torch.nn.Transformer module for machine translation on the Multi30k dataset. It performs pretty well (32.2 Bleu score) but I looked at scaling up ...
Matt Harrison's user avatar
2 votes
1 answer
201 views

What is the basic difference between NAS, Hyperparameter Optimization/Tuning and Pruning?

As far as I understand: NAS = an algorithm that searches for the best NN architecture, i.e., how many layers, what activation function to use, how many neurons, etc. Hyperparameter Optim = finding ...
user366312's user avatar
0 votes
1 answer
75 views

L2 regularization in BN layers, how to set gamma?

I have read tensorflow's documents about batch normzalization , but still don't get what is the gamma regulizer? the link to document: https://www.tensorflow.org/api_docs/python/tf/keras/layers/...
sara yaghoobi's user avatar
1 vote
1 answer
110 views

Are there any guidelines on picking hyperparameters for Deep Reinforcement Learning?

I am trying to learn machine learning from Andrew NG's Machine learning specialization on Coursera. In the chapter about reinforcement learning Andrew NG said that if you do not select correct ...
EmperorAurelian's user avatar
1 vote
1 answer
85 views

Are there techniques for estimating optimal neural network size?

Are there techniques for estimating optimal neural network size? To replicate "AND gate", one does not need 1e1000 nodes in hidden layer. What would be the metric hinting at "too much ...
IndustryUser1942's user avatar
0 votes
2 answers
760 views

What should I do if my validation score is good, but my test score is bad?

I've trained my artificial neural network, and, as per standard practice, I've picked out the one neural network throughout training that did the best on my validation dataset. That is, the neural ...
Pro Q's user avatar
  • 103
0 votes
0 answers
20 views

How to speed up the learning process

I have built a network that performans pretty well on my data. The issue I have is that for a larger number of epochs at the start of the training process the val/train acc/loss are stagnating (for ...
Skobo Do's user avatar
1 vote
0 answers
33 views

How to make my neural networks designs more robust

Whenever, I design a neural network to solve a novel problem (requires a novel loss function i.e. not image classification) it always ends up being hypersensitive to batch size and learning rate. ...
Tom Huntington's user avatar
0 votes
1 answer
708 views

Why does GridSearchCV model give worse results despite same parameters used with base model

I am trying to make prediction using random forest regression and then utilize GridSearchCV to tune hyperparameters(just 'n_estimators'). However results of GridSearchCV are worse than base model. ...
dancineer's user avatar
1 vote
1 answer
115 views

Hyperparameter tuning methods for neural networks

I have a fully connected feedforward classifier neural network that uses the leaky ReLU activation function. I would like to apply a state-of-the-art hyperparameter tuning method to my methodology. ...
mdslt's user avatar
  • 111
0 votes
1 answer
78 views

Cross Validation and hyperparameter selection correct procedure

I am trying to run a regression supervised learning problem. The dataset is not very large and I wanted to do some cross-validation to avoid overfitting. As I have read it's important to do a ...
metc's user avatar
  • 1
1 vote
2 answers
62 views

How to manually adjust output from model? [closed]

I wonder if it is possible to add manual inference to the output of a model? For example, I have a model called 'net', and the output value of 'net' is a vector called v = [v1, ... vn]. v is a binary ...
Edee's user avatar
  • 111
3 votes
0 answers
37 views

How should I compare multiple machine learning models to be (generally) fair to all models?

I am testing multiple models on IBM HR Analytics Attrition Dataset (1470 lines) and HR Analytics dataset (15000 lines) for a research project. The models include traditional models (Naive Bayes, SVM), ...
Đào Minh Dũng's user avatar
2 votes
3 answers
230 views

Is it possible to learn the number of layers?

Is it possible, in a transformer or other deep architecture, to include the number of layers as a parameter of the model so it could be learned? In fact, I have a keras layer that I use to change the ...
arivero's user avatar
  • 51
0 votes
0 answers
28 views

Resolving Derivation Discrepancies for Differentiating through Optimization Paths

I'm reading the paper "Optimizing Millions of Hyperparameters by Implicit Differentiation". The key contribution of the paper is to show that you can replace optimizing through the ...
Decadz's user avatar
  • 1
3 votes
1 answer
705 views

How do I design the network for Deep Q-Network?

I am playing with deep q-learning and I am thinking about what the proper architecture of a network used for deep q-learning is. I have a very simple environment, basically a 18x18 matrix, where 3 ...
Joysn's user avatar
  • 133
1 vote
1 answer
241 views

How can I interpret the value returned by score(X) method of sklearn.neighbors.KernelDensity?

For sklearn.neighbors.KernelDensity, its score(X) method according to the sklearn KDE documentation says: Compute the log-...
Arun's user avatar
  • 235
1 vote
1 answer
591 views

How does stochastic gradient descent undo the normalization done by the batch normalization?

I want to understand the handshake between SGD (or mini-batch GD) and batch normalization. Below, an explanation quoted from this Medium article. However, I am confused about the denormalization by ...
Arighna's user avatar
  • 13
1 vote
2 answers
958 views

Random forests - are more estimators always better?

I'm learning about more advanced methods of hyperparameter optimization, such as the Bayesian methods in the scikit-optimize package. For those unfamiliar with the ...
SuperCodeBrah's user avatar
2 votes
1 answer
101 views

What components of reinforcement learning influence the result the most?

I'm working on my thesis concerning a reinforcement learning problem and am trying to prioritise my time on different components of it: Formalising the agent environment (like the design of state-, ...
kitaird's user avatar
  • 115
0 votes
0 answers
32 views

Is there a way to adapt Particle Swarm Optimization to an incremental/online learning setting?

As stated in the title, is there a way to adapt PSO to an online scenario where new data samples arrive continuously? In more detail: suppose that I have a classifier with several parameters for which ...
Elise Le's user avatar
0 votes
2 answers
5k views

How to fine-tune GPT-J with small dataset

I have followed this guide as closely as possible: https://github.com/kingoflolz/mesh-transformer-jax I'm trying to fine-tune GPT-J with a small dataset of ~500 lines: ...
Ilya Karnaukhov's user avatar
5 votes
2 answers
763 views

Do we need automatic hyper-parameter tuning when we have a large enough dataset?

Hyperparameter tuning is the process of selecting the optimal hyperparameters for an ANN. Now, my guess is that, if we have sufficient data (say, 1.4 million for, say, 6 features), the model can be ...
user366312's user avatar
3 votes
1 answer
313 views

How do I choose the hyper-parameters for a model to detect different guitar chords?

I need to build a hand detector that recognizes the chord played by a hand on a guitar. I read this article Static Hand Gesture Recognition using Convolutional Neural Network with Data Augmentation ...
Deffo's user avatar
  • 83
1 vote
0 answers
35 views

How do you decide that you have tested enough hyper-parameter combinations for a specific neural network architecture?

How do you decide that you have tested enough hyper-parameter combinations for a specific neural network architecture to discard it and move on to a new model? Do you have a structured (generic) ...
DanDan's user avatar
  • 11
2 votes
0 answers
101 views

Has the idea of using different learning rates for different layers been explored in the literature?

I wonder whether there are heuristic rules for the optimal selection of learning rates for different layers. I expect that there is no general recipe, but probably there are some choices that may be ...
spiridon_the_sun_rotator's user avatar
1 vote
1 answer
134 views

Can the optimal learning rate differ for different architectures?

In several courses and tutorials about neural networks, people often say that the learning rate (LR) should be the first hyper-parameter to be tuned before we tweak the others. For example, in this ...
GKozinski's user avatar
  • 1,270
1 vote
0 answers
68 views

Why doesn't anyone use reinforcement learning to find the best possible alternative to backpropagation?

To be clear, I'm very uninformed on the topic of alternative learning algorithms to backprop, all my knowledge comes from articles like these: lets-not-stop-at-backprop backprop-alternatives we-need-a-...
Ethan's user avatar
  • 103
9 votes
3 answers
20k views

How to determine the embedding size?

When we are training a neural network, we are going to determine the embedding size to convert the categorical (in NLP, for instance) or continuous (in computer vision or voice) information to hidden ...
Lerner Zhang's user avatar
0 votes
1 answer
1k views

Is it valid to implement hyper-parameter tuning and THEN cross-validation?

I have a multi-label classification task I am solving. I have done hyperparameter tuning (with Keras Tuner) to determine the best configuration for my neural network. Is it valid to do this (determine ...
user9317212's user avatar
1 vote
1 answer
186 views

How many singular vectors do we need to calculate for SVD?

In the geometrical interpretation of SVD, the data points that we have need to be imagined as points in high dimensional space (say $d$-dimensional space). But we need to find a hyperplane in $k-$...
hanugm's user avatar
  • 3,890
1 vote
0 answers
39 views

Are there guiding principles as to which activation functions suit a given RL algorithm?

Are there rules of thumb as to which activation functions work well (or which one would not) on the policy and value network of a class of RL algorithms? For hidden layers and for the output layer. ...
mugoh's user avatar
  • 531
6 votes
2 answers
4k views

What are the best hyper-parameters to tune in reinforcement learning?

Obviously, this is somewhat subjective, but what hyper-parameters typically have the most significant impact on an RL agent's ability to learn? For example, the replay buffer size, learning rate, ...
Dylan Kerler's user avatar
2 votes
0 answers
53 views

Is there an optimal number of species for NEAT?

Is there an optimal number of species for NEAT? Since too low and too high is bad, I am thinking about adjusting the threshold of the distance function at runtime in order to have the number of ...
IAmUser's user avatar
  • 51
1 vote
0 answers
371 views

Why does Adam optimizer work slower than Adagrad, Adadelta, and SGD for Neural Collaborative Filtering (NCF)?

I've been working on Neural Collaborative Filtering (NCF) recently to build a recommender system using Tensorflow Recommenders. Doing some hyperparameter tuning with different optimizers available in ...
bkaankuguoglu's user avatar
3 votes
1 answer
113 views

What is the most statistically acceptable method for tuning neural network hyperparameters on very small datasets?

Neural networks are usually evaluated by dividing a dataset into three splits: training, validation, and test The idea is that critical hyperparameters of the network such as the number of epochs ...
Mike NZ's user avatar
  • 411
0 votes
2 answers
971 views

Do larger numbers of hidden layers have a bigger effect on a classification model's accuracy?

I trained different classification models using Keras with different numbers of hidden layers and the same number of neurons in each layer. What I found was the accuracy of the models decreased as the ...
Shonix3373's user avatar
3 votes
0 answers
538 views

How to Select Model Parameters for Transformer (Heads, number of layers, etc)

Is there a general guideline on how the Transformer model parameters should be selected, or the range of these parameters that should be included in a hyperparameter sweep? Number of heads Number of ...
Athena Wisdom's user avatar
0 votes
1 answer
946 views

What to do with a GAN that trained well but got worse over time?

I am training a WGAN-GP network based on the following paper, though I am using a different dataset. Now, for the first ~ 60-70 epochs, my network trained really well, which I could see in the loss ...
Anonymous5638's user avatar
2 votes
1 answer
56 views

Is it possible to train one part of the network with a particular learning rate and the other part with a different one?

I have a combined network consisting of two parts: one is for images and the other is for numerical data. Each sample is matched with a numerical case by an ID. For this combined network, a ...
bit_scientist's user avatar
4 votes
1 answer
152 views

Bayesian hyperparameter optimization, is it worth it?

In the Deep Learning book by Goodfellow et al., section 11.4.5 (p. 438), the following claims can be found: Currently, we cannot unambiguously recommend Bayesian hyperparameter optimization as an ...
Stefano Barone's user avatar
3 votes
1 answer
4k views

What should the value of epsilon be in the Q-learning?

I am trying to understand Reinforcement Learning and already explored different Youtube videos, blog posts, and Wikipedia articles. What I don't understand is the impact of $\epsilon$. What value ...
Exploring's user avatar
  • 343
2 votes
1 answer
1k views

What is the impact of changing the crossover and mutation rates?

What is the impact of using a: low crossover rate high crossover rate low mutation rate high mutation rate
fathese's user avatar
  • 131
1 vote
0 answers
558 views

Why does PPO lead to a worse performance than TRPO in the same task?

I am training an agent with an Actor-Critic network and update it with TRPO so far. Now, I tried out PPO and the results are drastically different and bad. I only changed from TRPO to PPO, the rest of ...
thsolyt's user avatar
  • 31
3 votes
1 answer
226 views

How should I change the hyper-parameters of the C51 algorithm, in order to obtain higher reward?

I have a scenario where, in an ideal situation, the greedy approach is the best, but when non-idealities are introduced which can be learned, DQN starts doing better. So, after checking what DQN ...
user3656142's user avatar