# Questions tagged [hyperparameter-optimization]

For questions related to the concept of hyper-parameter optimization, that is, the task of finding the best hyper-parameters for a particular learning algorithm (e.g. gradient descent) or model (e.g. a multi-layer neural network) using an optimization method (e.g. Bayesian optimization or genetic algorithms).

44 questions
Filter by
Sorted by
Tagged with
19 views

### How do we choose the filters for the convolutional layer of a convolution neural network?

Since the hidden layers of a CNN work as a trainable feature extractor, more detailed content based on a larger number of pixels shall require bigger filter sizes. But for cases where localized ...
19 views

### Should we start with a small batch-size and increase during training to improve sample efficiency?

Just made an interesting observation playing around with the stable-baseline's implementation of PPO and the BipedalWalker environment from OpenAI's Gym. But I believe this should be a general ...
11 views

### Number of parameters in Resnet-50 [migrated]

I'm using Keras, and I am struggling to know how many parameters Resnet-50 has. Keras documentation says around 25M, while if I ...
29 views

### Is it theoretically possible (or impossible) that principal component analysis worsens the performance of the model?

In case I had a prediction model and decided to add a PCA step prior to the model, is it theoretically possible/impossible that the number of output dimensions that is better for all tests may perform ...
12 views

### Larger neural network with almost similar training and validation error as smaller network

I am building a neural network that takes as input 202 units and outputs a 200 dimension continuous variable. While trying to find the best model, one of the parameters i tune is the the number of ...
18 views

### How to know if the hyperparameters of a neural network relate to each other?

According this thread some hyperparameters are independent from each other while some are directly related. One of the answers give an example where two hyperparameters affect each other. For ...
19 views

### How is a validation set used to tune the hyperparameters in a non-biased way, if the new models depends on the values of these?

I've built a neural network from the scratch, choosing arbitrary numbers for the hyperparameters: learning rate, number of hidden layers and neurons for these, number of epochs and size of mini ...
43 views

### After having selected the best model with cross-validation, for how long should I train it?

When using k-fold cross-validation in a deep learning problem, after you have computed your hyper-parameters, how do you decide how long to train your final model? My understanding is that, after the ...
34 views

### Why can't I Hyper tune my KNNBasic Algorithm?

I've been trying to hyper tuning my KNNBasic algorithm by the help of grid search for recommendation system for movie review data. The problem is that both of my KNNBasicTuned and KNNBasicUntuned ...
93 views

### Why do we need both the validation set and test set?

I know that this has been asked a hundred times before, however, I was not able to find a question (and an answer) which actually answered what I wanted to know, respectively, which explained it in a ...
72 views

### Are there principled ways of tuning a neural network in case of overfitting and underfitting?

Whenever I tune my neural network, I usually take the common approach of defining some layers with some neurons. If it overfits, I reduce the layers, neurons, add dropout, utilize regularisation. ...
49 views

### Why do momentum techniques not work well for RNNs?

AFAIK, momentum is quite useful when training CNNs, and can speed-up the training substantially without any drop in validation accuracy. I've recently learned that it is not as helpful for RNNs ...
238 views

### What kind of optimizer is suggested to use for binary classification of similar images?

I have spent some time searching Google and wasn't able to find out what kind of optimization algorithm is best for binary classification when images are similar to one another. I'd like to read ...
15 views

### When stacking LSTM's, should the hidden units increase?

I'm using Weights and Biases to do some hyperparameter sweeping for a supervised sequence-to-sequence problem I'm working on. One thing I noticed is that the sweeps with a gradually increasing number ...
14 views

### augmentation, cross-validation, test evaluation

I have, say, a (balanced) data-set with 2k images for binary classification. What I have done is that randomly divided the data-set into 5 folds; copy-pasted all 5-fold data-set to have 5 exact ...
31 views

### Are there optimizers that schedule their learning rate, momentum etc. autonomously?

I'm aware there are some optimizer such as Adam that adjust the learning rate for each dimension during training. However, afaik, the maximum learning rate they can have is still determined by the ...
53 views

### Is there a reason to choose regular momentum over Nesterov momentum for neural networks?

I've been reading about Nesterov momentum from here and it seems like a nice improvement over regular momentum with no extra cost whatsoever. However, is this really the case? Are there instances ...
38 views

### When training a CNN, what are the hyperparameters to tune first?

I am training a convolutional neural network for object detection. Apart from the learning rate, what are the other hyperparameters that I should tune? And in what order of importance? Besides, I read ...
36 views

### What are some ways to quickly evaluate the potential of a given NN architecture?

Main question Is there some way we can leverage general knowledge of how certain hyperparameters affect performance, to very rapidly get some sort of estimate for how good a given architecture could ...
15 views

### Is it normal to see oscillations in tested hyperparameters during bayesian optimisation?

I've been trying out bayesian hyperparameter optimisation (with TPE) on a simple CNN applied to the MNIST handwritten digit dataset. I noticed that over iterations of the optimisation loop, the tested ...
34 views

### How can I do hyperparameter optimization for a CNN-LSTM neural network?

I have built a CNN-LSTM neural network with 2 inputs and 2 outputs in Keras. I trained the network with model.fit_generator() (and not ...
21 views

### Hyperparameter optimisation over entire range or shorter range of training episodes in Deep Reinforcement Learning

I am optimising hyperparameters for my deep reinforcement learning project (using PPO2, DQN and A2C) and was wondering: Should I find the optimum hyperparameters to get maximum reward from training ...
17 views

### How to organize model training hyperparameters

I am working on multiple deep learning projects, most of them in the area of computer vision. For many of them I create multiple models, try different approaches, use various model architectures. And ...
732 views

### What is a “surrogate model”?

In the following paragraph from the book Automated Machine Learning: Methods, Systems, Challenges (by Frank Hutter et al.) In this section we first give a brief introduction to Bayesian ...
48 views

### How can I avoid overfitting when doing parameter tuning?

I very often applied a grid search to tune the parameters of my supervised model. I have the feeling that parameter tuning will eventually (very often) lead to overfitting? Is this crazy to say? Is ...
48 views

### How to use a deep learning network on new data-set?

I am trying to use a network for classification. This network works very well on the author's example data, but doesn't work on new data. Currently, I am using the popular EEG Motor Movement/Imagery ...
14 views

### Choosing best combinations from all possible combination expressions based few variables, unary operators, binary operators

I have a few financial variables of a stock universe like OHLC prices, volume, and other fundamentals with varying time-frequency. Using this set I'm creating an expression that gives the weights to ...
42 views

### How can we conclude that an optimization algorithm is better than another one

When we test a new optimization algorithm, what the process that we need to do?For example, do we need to run the algorithm several times, and pick a best performance,i.e., in terms of accuracy, f1 ...
105 views

### Threshold selection for Siamese network hyper-parameter tuning

I'm interested in modeling a Siamese network for facial verification. I've already written a simple working model that inputs feature vectors generated from two CNNs with shared weights then outputs a ...
20 views

### Improving Recall of a Certain Class

Let's say that we have a test data set with $20,000$ observations for which we want to make a binary prediction for. When we apply our best trained model to this data set (e.g. logistic regression ...
135 views

It (Adagrad) adapts the learning rate to the parameters, performing smaller updates (i.e. low learning rates) for parameters associated with frequently occurring features, and larger updates (i.e. ...
111 views

### What is the best measure against overfitting?

I wanted to ask about the methodology of testing the ML models against overfitting. Please note that I don't mean any overfitting reducing methods like regularisation, just a measure to judge whether ...
33 views

### How do you scale your ML problems?

While I have limited resource usually to train my machine learning models, I often find that my hyperparameter optimization procedure is not necessary using all my GPU and CPU, and that is because the ...
49 views

### Does this hyperparameter optimisation approach yield the optimal hyperparameters?

Say I have a ML model which is not very costly to train. It has around say 5 hyperparameters. One way to select best hyperparameters would be to keep all the other hyperparamaters fixed and train ...
73 views

### Can we automate the choice of the hyper-parameters of the evolutionary algorithms?

Certain hyper-parameters (e.g. the size of the offspring generation or the definition of the fitness function) and the design (e.g. how the mutation is performed) of evolutionary algorithms usually ...
65 views

### Why doesnt my lstm model for time series prediction improve after certain level of performance?

I created an lstm model which predicts multioutput sequeances. It takes variable length sequences as input. These sequences are padded with zero to obtain equal length. Note that the time series are ...
42 views

### Evolving Machine Learning

It seems to me that, right now, the key to making a good Machine Learning model is in choosing the right combination of hyper-parameters. Firstly: Am I right in saying, if a model is able to tune it'...
50 views

### How to use TensorFlow with hyperparameter tuning to optimize parameters for a robot simulator

I am trying to implement a DNN to optimize a set of 7 parameters that are used in a robot swarm simulator on the ARGoS platform. the program is a compiled C++ executable that reads the parameters from ...
188 views

### How is neural architecture search performed?

I have come across something that IBM offers called neural Architecture search. You feed it a data set and it outputs an initial neural Architecture that you can train. How is neural architecture ...
60 views

### Are there Python packages for random search hyper-parameter optimisation?

Which Python packages do you recommend for random search hyperparameter optimization to use? Is there any recent and good one (better than the one in scikit-learn)?
72 views

### Are there Python packages for recent Bayesian optimization methods?

I want to try and compare different optimization methods in some datasets. I know that in scikit-learn there are some corresponding functions for the grid and random search optimizations. However, I ...
289 views

### How do I choose the size of the hidden state of a GRU?

I'm trying to understand how the size of the hidden state affects the GRU. For example, suppose I want to make a GRU count. I'm gonna feed it with three numbers, and I expect it to predict the ...