Questions tagged [hyperparameter-optimization]

For questions related to the concept of hyper-parameter optimization, that is, the task of finding the best hyper-parameters for a particular learning algorithm (e.g. gradient descent) or model (e.g. a multi-layer neural network) using an optimization method (e.g. Bayesian optimization or genetic algorithms).

40 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
14 votes
1 answer
248 views

Will parameter sweeping on one split of data followed by cross validation discover the right hyperparameters?

Let's call our dataset splits train/test/evaluate. We're in a situation where we require months of data. So we prefer to use the evaluation dataset as infrequently as possible to avoid polluting our ...
user avatar
6 votes
1 answer
323 views

An intuitive explanation of Adagrad, its purpose and its formula

It (Adagrad) adapts the learning rate to the parameters, performing smaller updates (i.e. low learning rates) for parameters associated with frequently occurring features, and larger updates (i.e. ...
user avatar
  • 103
4 votes
1 answer
593 views

When training a CNN, what are the hyperparameters to tune first?

I am training a convolutional neural network for object detection. Apart from the learning rate, what are the other hyperparameters that I should tune? And in what order of importance? Besides, I read ...
user avatar
  • 41
3 votes
0 answers
15 views

How should I compare multiple machine learning models to be (generally) fair to all models?

I am testing multiple models on IBM HR Analytics Attrition Dataset (1470 lines) and HR Analytics dataset (15000 lines) for a research project. The models include traditional models (Naive Bayes, SVM), ...
user avatar
3 votes
0 answers
87 views

Are there principled ways of tuning a neural network in case of overfitting and underfitting?

Whenever I tune my neural network, I usually take the common approach of defining some layers with some neurons. If it overfits, I reduce the layers, neurons, add dropout, utilize regularisation. ...
user avatar
  • 151
3 votes
0 answers
47 views

What are some ways to quickly evaluate the potential of a given NN architecture?

Main question Is there some way we can leverage general knowledge of how certain hyperparameters affect performance, to very rapidly get some sort of estimate for how good a given architecture could ...
user avatar
3 votes
0 answers
66 views

How can I do hyperparameter optimization for a CNN-LSTM neural network?

I have built a CNN-LSTM neural network with 2 inputs and 2 outputs in Keras. I trained the network with model.fit_generator() (and not ...
user avatar
  • 31
3 votes
0 answers
26 views

How to organize model training hyperparameters

I am working on multiple deep learning projects, most of them in the area of computer vision. For many of them I create multiple models, try different approaches, use various model architectures. And ...
user avatar
  • 155
2 votes
0 answers
37 views

Has the idea of using different learning rates for different layers been explored in the literature?

I wonder whether there are heuristic rules for the optimal selection of learning rates for different layers. I expect that there is no general recipe, but probably there are some choices that may be ...
user avatar
2 votes
0 answers
29 views

Is there an optimal number of species for NEAT?

Is there an optimal number of species for NEAT? Since too low and too high is bad, I am thinking about adjusting the threshold of the distance function at runtime in order to have the number of ...
user avatar
  • 51
2 votes
0 answers
58 views

How to Select Model Parameters for Transformer (Heads, number of layers, etc)

Is there a general guideline on how the Transformer model parameters should be selected, or the range of these parameters that should be included in a hyperparameter sweep? Number of heads Number of ...
user avatar
2 votes
0 answers
25 views

Deriving hyperparameter updates in Online Interactive Collaborative Filtering

I've been going through "Online Interactive Collaborative Filtering Using Multi-Armed Bandit with Dependent Arms" by Wang et al. and am unable to understand how the update equations for the ...
user avatar
2 votes
0 answers
25 views

Is it normal to see oscillations in tested hyperparameters during bayesian optimisation?

I've been trying out bayesian hyperparameter optimisation (with TPE) on a simple CNN applied to the MNIST handwritten digit dataset. I noticed that over iterations of the optimisation loop, the tested ...
user avatar
1 vote
0 answers
28 views

How do you decide that you have tested enough hyper-parameter combinations for a specific neural network architecture?

How do you decide that you have tested enough hyper-parameter combinations for a specific neural network architecture to discard it and move on to a new model? Do you have a structured (generic) ...
user avatar
  • 11
1 vote
0 answers
52 views

Why doesn't anyone use reinforcement learning to find the best possible alternative to backpropagation?

To be clear, I'm very uninformed on the topic of alternative learning algorithms to backprop, all my knowledge comes from articles like these: lets-not-stop-at-backprop backprop-alternatives we-need-a-...
user avatar
  • 103
1 vote
1 answer
75 views

How many singular vectors do we need to calculate for SVD?

In the geometrical interpretation of SVD, the data points that we have need to be imagined as points in high dimensional space (say $d$-dimensional space). But we need to find a hyperplane in $k-$...
user avatar
  • 3,099
1 vote
0 answers
21 views

Are there guiding principles as to which activation functions suit a given RL algorithm?

Are there rules of thumb as to which activation functions work well (or which one would not) on the policy and value network of a class of RL algorithms? For hidden layers and for the output layer. ...
user avatar
  • 509
1 vote
0 answers
36 views

Why does Adam optimizer work slower than Adagrad, Adadelta, and SGD for Neural Collaborative Filtering (NCF)?

I've been working on Neural Collaborative Filtering (NCF) recently to build a recommender system using Tensorflow Recommenders. Doing some hyperparameter tuning with different optimizers available in ...
user avatar
1 vote
0 answers
193 views

Why does PPO lead to a worse performance than TRPO in the same task?

I am training an agent with an Actor-Critic network and update it with TRPO so far. Now, I tried out PPO and the results are drastically different and bad. I only changed from TRPO to PPO, the rest of ...
user avatar
  • 31
1 vote
0 answers
31 views

Best/quickest approach for tuning the hyperparameters of a restricted boltzmann machine

I have an RBM model which takes extremely long to train and evaluate because of the large number of free parameters and the large amount of input data. What would be the most efficient way of tuning ...
user avatar
1 vote
0 answers
38 views

How many training runs are needed to obtain a credible value for performance?

I'm trying to optimize a neural network. For that, I'm changing parameters like the batch size, learning rate, weight initialization, etc. A neural network is not a deterministic algorithm, so, in ...
user avatar
1 vote
0 answers
102 views

How do we choose the filters for the convolutional layer of a convolution neural network?

Since the hidden layers of a CNN work as a trainable feature extractor, more detailed content based on a larger number of pixels shall require bigger filter sizes. But for cases where localized ...
user avatar
  • 1,193
1 vote
0 answers
47 views

Should we start with a small batch-size and increase during training to improve sample efficiency?

Just made an interesting observation playing around with the stable-baseline's implementation of PPO and the BipedalWalker environment from OpenAI's Gym. But I believe this should be a general ...
user avatar
1 vote
0 answers
113 views

Why can't I Hyper tune my KNNBasic Algorithm?

I've been trying to hyper tuning my KNNBasic algorithm by the help of grid search for recommendation system for movie review data. The problem is that both of my KNNBasicTuned and KNNBasicUntuned ...
user avatar
1 vote
0 answers
26 views

When stacking LSTM's, should the hidden units increase?

I'm using Weights and Biases to do some hyperparameter sweeping for a supervised sequence-to-sequence problem I'm working on. One thing I noticed is that the sweeps with a gradually increasing number ...
user avatar
  • 121
1 vote
0 answers
44 views

Are there optimizers that schedule their learning rate, momentum etc. autonomously?

I'm aware there are some optimizer such as Adam that adjust the learning rate for each dimension during training. However, afaik, the maximum learning rate they can have is still determined by the ...
user avatar
1 vote
0 answers
48 views

Hyperparameter optimisation over entire range or shorter range of training episodes in Deep Reinforcement Learning

I am optimising hyperparameters for my deep reinforcement learning project (using PPO2, DQN and A2C) and was wondering: Should I find the optimum hyperparameters to get maximum reward from training ...
user avatar
  • 11
1 vote
0 answers
391 views

Threshold selection for Siamese network hyper-parameter tuning

I'm interested in modeling a Siamese network for facial verification. I've already written a simple working model that inputs feature vectors generated from two CNNs with shared weights then outputs a ...
user avatar
  • 11
1 vote
0 answers
25 views

Improving Recall of a Certain Class

Let's say that we have a test data set with $20,000$ observations for which we want to make a binary prediction for. When we apply our best trained model to this data set (e.g. logistic regression ...
user avatar
  • 21
1 vote
0 answers
102 views

Why doesnt my lstm model for time series prediction improve after certain level of performance?

I created an lstm model which predicts multioutput sequeances. It takes variable length sequences as input. These sequences are padded with zero to obtain equal length. Note that the time series are ...
user avatar
  • 111
0 votes
0 answers
11 views

Finding "look_back" & "look_ahead" hyper-parameters for Seq2Seq models

For Seq2Seq deep learning architectures, viz., LSTM/GRU and multivariate, multistep time series forecasting, it is important to convert the data to a 3D dimension: (batch_size, look_back, ...
user avatar
  • 185
0 votes
0 answers
11 views

What is a good first guess to set hyper parameters for Adam optimiser for Faster RCNN?

Is there a way to make an educated guess on how to set the hyper parameters for Adam optimiser for Faster RCNN depending on the size of the dataset and the loss function? I use the default values for ...
user avatar
0 votes
0 answers
27 views

Resolving Derivation Discrepancies for Differentiating through Optimization Paths

I'm reading the paper "Optimizing Millions of Hyperparameters by Implicit Differentiation". The key contribution of the paper is to show that you can replace optimizing through the ...
user avatar
  • 1
0 votes
0 answers
29 views

Should hyper-parameters used in the mutation operator be fixed or variable?

Let's say I have a function $f(p_{i,t})$ and want to optimize $p_{i,t}$ so that I minimize a certain loss. Here, $p_{i,t}$ is the individual $i$ at iteration $t$. Now, $p_{i,t}$ can live anywhere ...
user avatar
  • 179
0 votes
0 answers
38 views

How to identify clusters of best hyperparameters from noisy hyperparameter optimization?

I'd like to optimize the hyperparameters for a random forest model that is already somewhat time-consuming to train. With adding cross-validation and multiple hyperparameter combinations, I might be ...
user avatar
0 votes
0 answers
21 views

Is there a way to adapt Particle Swarm Optimization to an incremental/online learning setting?

As stated in the title, is there a way to adapt PSO to an online scenario where new data samples arrive continuously? In more detail: suppose that I have a classifier with several parameters for which ...
user avatar
0 votes
1 answer
854 views

How to fine-tune GPT-J with small dataset

I have followed this guide as closely as possible: https://github.com/kingoflolz/mesh-transformer-jax I'm trying to fine-tune GPT-J with a small dataset of ~500 lines: ...
user avatar
0 votes
0 answers
12 views

Loss & accuracy curves from learning rate range test interpretation

I am working on a project doing experiments with the Learning Rate Range Test (See "A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and ...
user avatar
0 votes
0 answers
18 views

How to manually optimize Neural Networks the most systematical way?

Do you have any ideas or guidance on how to do manual neural network optimization in the most systematic way? Especially when models train longer and the effects of hyperparameter fitting are very ...
user avatar
0 votes
0 answers
44 views

Unable to meet desired mean squared error

I wish to get MSE < 0.5 on test data (https://easyupload.io/zr7xf3) which is 20% of given data chosen randomly. But I am reaching 0.73 using both plain Ridge Regression as well as a neural network ...
user avatar