Questions tagged [hyperparameter-optimization]

For questions related to the concept of hyper-parameter optimization, that is, the task of finding the best hyper-parameters for a particular learning algorithm (e.g. gradient descent) or model (e.g. a multi-layer neural network) using an optimization method (e.g. Bayesian optimization or genetic algorithms).

Filter by
Sorted by
Tagged with
16 votes
1 answer
375 views

Will parameter sweeping on one split of data followed by cross validation discover the right hyperparameters?

Let's call our dataset splits train/test/evaluate. We're in a situation where we require months of data. So we prefer to use the evaluation dataset as infrequently as possible to avoid polluting our ...
0 votes
0 answers
11 views

Optuna Hyperband Algorithm Not Following Expected Model Training Scheme

I have observed an issue while using the Hyperband algorithm in Optuna. According to the Hyperband algorithm, when min_resources = 5, max_resources = 20, and reduction_factor = 2, the search should ...
0 votes
1 answer
73 views

L2 regularization in BN layers, how to set gamma?

I have read tensorflow's documents about batch normzalization , but still don't get what is the gamma regulizer? the link to document: https://www.tensorflow.org/api_docs/python/tf/keras/layers/...
0 votes
2 answers
5k views

How to fine-tune GPT-J with small dataset

I have followed this guide as closely as possible: https://github.com/kingoflolz/mesh-transformer-jax I'm trying to fine-tune GPT-J with a small dataset of ~500 lines: ...
0 votes
1 answer
637 views

Why does GridSearchCV model give worse results despite same parameters used with base model

I am trying to make prediction using random forest regression and then utilize GridSearchCV to tune hyperparameters(just 'n_estimators'). However results of GridSearchCV are worse than base model. ...
0 votes
1 answer
72 views

Cross Validation and hyperparameter selection correct procedure

I am trying to run a regression supervised learning problem. The dataset is not very large and I wanted to do some cross-validation to avoid overfitting. As I have read it's important to do a ...
0 votes
0 answers
17 views

Output Not Changing - Feeding Previous Outputs Back Into a Model

Full disclosure, I also posted this on Stack Overflow I have put a more theory based bent towards the question itself here I have a simple model in pytorch based on the quickstart except instead of a ...
1 vote
1 answer
103 views

How to fairly conduct a model performance with 5-fold cross validation after augmentation?

I have, say, a (balanced) data-set with 2k images for binary classification. What I have done is that randomly divided the data-set into 5 folds; copy-pasted all 5-fold data-set to have 5 exact ...
7 votes
3 answers
2k views

When training a CNN, what are the hyperparameters to tune first?

I am training a convolutional neural network for object detection. Apart from the learning rate, what are the other hyperparameters that I should tune? And in what order of importance? Besides, I read ...
8 votes
1 answer
513 views

An intuitive explanation of Adagrad, its purpose and its formula

It (Adagrad) adapts the learning rate to the parameters, performing smaller updates (i.e. low learning rates) for parameters associated with frequently occurring features, and larger updates (i.e. ...
1 vote
0 answers
34 views

A Quest for Maximizing Expected Value: Optimizing Algorithmic Trading Parameters

I am engaged in algorithmic trading, employing specialized models that utilize various parameters to signal trading opportunities. Once I get a signal, I could execute a trade and everything is ...
0 votes
0 answers
12 views

Incorporating HiPlot and Keras

I just started to learn about Keras and train some models, and I came across HiPlot which is used for tuning hyperparameters. I was wondering if HiPlot can also be used to see what parameters would ...
0 votes
0 answers
25 views

Longer DNN training times when using evolutionary algorithms

I am comparing my deep neural network (DNN) performance when using 2 types of optimizers: gradient-based Adam (properly tuned) and a population-based optimization algorithm (e.g., genetic algorithm (...
1 vote
0 answers
81 views

Why would increasing layers in PyTorch Transformer significantly increase loss?

I have a simple torch.nn.Transformer module for machine translation on the Multi30k dataset. It performs pretty well (32.2 Bleu score) but I looked at scaling up ...
2 votes
1 answer
183 views

What is the basic difference between NAS, Hyperparameter Optimization/Tuning and Pruning?

As far as I understand: NAS = an algorithm that searches for the best NN architecture, i.e., how many layers, what activation function to use, how many neurons, etc. Hyperparameter Optim = finding ...
1 vote
2 answers
892 views

Random forests - are more estimators always better?

I'm learning about more advanced methods of hyperparameter optimization, such as the Bayesian methods in the scikit-optimize package. For those unfamiliar with the ...
1 vote
1 answer
108 views

Are there any guidelines on picking hyperparameters for Deep Reinforcement Learning?

I am trying to learn machine learning from Andrew NG's Machine learning specialization on Coursera. In the chapter about reinforcement learning Andrew NG said that if you do not select correct ...
1 vote
1 answer
82 views

Are there techniques for estimating optimal neural network size?

Are there techniques for estimating optimal neural network size? To replicate "AND gate", one does not need 1e1000 nodes in hidden layer. What would be the metric hinting at "too much ...
0 votes
2 answers
635 views

What should I do if my validation score is good, but my test score is bad?

I've trained my artificial neural network, and, as per standard practice, I've picked out the one neural network throughout training that did the best on my validation dataset. That is, the neural ...
0 votes
0 answers
20 views

How to speed up the learning process

I have built a network that performans pretty well on my data. The issue I have is that for a larger number of epochs at the start of the training process the val/train acc/loss are stagnating (for ...
3 votes
1 answer
218 views

How should I change the hyper-parameters of the C51 algorithm, in order to obtain higher reward?

I have a scenario where, in an ideal situation, the greedy approach is the best, but when non-idealities are introduced which can be learned, DQN starts doing better. So, after checking what DQN ...
1 vote
0 answers
31 views

How to make my neural networks designs more robust

Whenever, I design a neural network to solve a novel problem (requires a novel loss function i.e. not image classification) it always ends up being hypersensitive to batch size and learning rate. ...
9 votes
3 answers
19k views

How to determine the embedding size?

When we are training a neural network, we are going to determine the embedding size to convert the categorical (in NLP, for instance) or continuous (in computer vision or voice) information to hidden ...
1 vote
1 answer
114 views

Hyperparameter tuning methods for neural networks

I have a fully connected feedforward classifier neural network that uses the leaky ReLU activation function. I would like to apply a state-of-the-art hyperparameter tuning method to my methodology. ...
1 vote
1 answer
182 views

How many singular vectors do we need to calculate for SVD?

In the geometrical interpretation of SVD, the data points that we have need to be imagined as points in high dimensional space (say $d$-dimensional space). But we need to find a hyperplane in $k-$...
1 vote
2 answers
60 views

How to manually adjust output from model? [closed]

I wonder if it is possible to add manual inference to the output of a model? For example, I have a model called 'net', and the output value of 'net' is a vector called v = [v1, ... vn]. v is a binary ...
3 votes
0 answers
35 views

How should I compare multiple machine learning models to be (generally) fair to all models?

I am testing multiple models on IBM HR Analytics Attrition Dataset (1470 lines) and HR Analytics dataset (15000 lines) for a research project. The models include traditional models (Naive Bayes, SVM), ...
2 votes
3 answers
229 views

Is it possible to learn the number of layers?

Is it possible, in a transformer or other deep architecture, to include the number of layers as a parameter of the model so it could be learned? In fact, I have a keras layer that I use to change the ...
0 votes
0 answers
28 views

Resolving Derivation Discrepancies for Differentiating through Optimization Paths

I'm reading the paper "Optimizing Millions of Hyperparameters by Implicit Differentiation". The key contribution of the paper is to show that you can replace optimizing through the ...
67 votes
4 answers
120k views

How to select number of hidden layers and number of memory cells in an LSTM?

I am trying to find some existing research on how to select the number of hidden layers and the size of these of an LSTM-based RNN. Is there an article where this problem is being investigated, i.e., ...
1 vote
1 answer
525 views

How to choose a suitable threshold value for the Shi-Tomasi corner detection algorithm?

While implementing the Shi-Tomasi corner detection algorithm, I got stuck in deciding a suitable threshold for corner detection. In the Shi-Tomasi algorithm, all those points that qualify $\min( \...
5 votes
1 answer
361 views

How do I design a neural network that breaks a 5-letter word into its corresponding syllables?

I am going to design a neural network which will be able to break a 5-letter word into its corresponding syllables (hybrid syllables, I mean it will not strictly adhere to grammatical syllable rules ...
3 votes
1 answer
616 views

How do I design the network for Deep Q-Network?

I am playing with deep q-learning and I am thinking about what the proper architecture of a network used for deep q-learning is. I have a very simple environment, basically a 18x18 matrix, where 3 ...
1 vote
1 answer
219 views

How can I interpret the value returned by score(X) method of sklearn.neighbors.KernelDensity?

For sklearn.neighbors.KernelDensity, its score(X) method according to the sklearn KDE documentation says: Compute the log-...
1 vote
1 answer
567 views

How does stochastic gradient descent undo the normalization done by the batch normalization?

I want to understand the handshake between SGD (or mini-batch GD) and batch normalization. Below, an explanation quoted from this Medium article. However, I am confused about the denormalization by ...
3 votes
2 answers
125 views

Does this hyperparameter optimisation approach yield the optimal hyperparameters?

Say I have a ML model which is not very costly to train. It has around say 5 hyperparameters. One way to select best hyperparameters would be to keep all the other hyperparamaters fixed and train ...
16 votes
2 answers
741 views

How can I automate the choice of the architecture of a neural network for an arbitrary problem?

Assume that I want to solve an issue with a neural network that either I can't fit to existing architectures (perceptron, Konohen, etc) or I'm simply not aware of the existence of those or I'm unable ...
2 votes
1 answer
101 views

What components of reinforcement learning influence the result the most?

I'm working on my thesis concerning a reinforcement learning problem and am trying to prioritise my time on different components of it: Formalising the agent environment (like the design of state-, ...
0 votes
1 answer
115 views

How can I systematically learn about the theory of neural networks?

I have seen a few articles about neural nets. Mostly they went along these lines: we tried these architectures, these meta parameters, we trained it for $x$ hours on $y$ CPUs, and it gave us these ...
0 votes
2 answers
105 views

Is it theoretically possible (or impossible) that principal component analysis worsens the performance of the model?

In case I had a prediction model and decided to add a PCA step prior to the model, is it theoretically possible/impossible that the number of output dimensions that is better for all tests may perform ...
8 votes
1 answer
3k views

How do we decide which membership function to use?

In classical set theory, there are two options for an element. It is either a member of a set or not. But in fuzzy set theory, there are membership functions to define the "rate" of an ...
2 votes
1 answer
655 views

How to design a neural network to predict the quadrant where a given point lies?

I've written a single perceptron that can predict whether a point is above or below a straight-line graph, given the correct training data and using a sign activation function. Now, I'm trying to ...
0 votes
1 answer
948 views

Is it valid to implement hyper-parameter tuning and THEN cross-validation?

I have a multi-label classification task I am solving. I have done hyperparameter tuning (with Keras Tuner) to determine the best configuration for my neural network. Is it valid to do this (determine ...
2 votes
0 answers
96 views

Has the idea of using different learning rates for different layers been explored in the literature?

I wonder whether there are heuristic rules for the optimal selection of learning rates for different layers. I expect that there is no general recipe, but probably there are some choices that may be ...
4 votes
1 answer
87 views

How do you efficiently choose the hyper-parameters of a neural network?

How do you efficiently choose the hyper-parameters of a neural network (e.g. the learning rate, number of layer, weights, etc.)?
0 votes
0 answers
31 views

Is there a way to adapt Particle Swarm Optimization to an incremental/online learning setting?

As stated in the title, is there a way to adapt PSO to an online scenario where new data samples arrive continuously? In more detail: suppose that I have a classifier with several parameters for which ...
5 votes
3 answers
363 views

How is neural architecture search performed?

I have come across something that IBM offers called neural architecture search. You feed it a data set and it outputs an initial neural architecture that you can train. How is neural architecture ...
5 votes
2 answers
761 views

Do we need automatic hyper-parameter tuning when we have a large enough dataset?

Hyperparameter tuning is the process of selecting the optimal hyperparameters for an ANN. Now, my guess is that, if we have sufficient data (say, 1.4 million for, say, 6 features), the model can be ...
3 votes
1 answer
300 views

How do I choose the hyper-parameters for a model to detect different guitar chords?

I need to build a hand detector that recognizes the chord played by a hand on a guitar. I read this article Static Hand Gesture Recognition using Convolutional Neural Network with Data Augmentation ...
1 vote
0 answers
35 views

How do you decide that you have tested enough hyper-parameter combinations for a specific neural network architecture?

How do you decide that you have tested enough hyper-parameter combinations for a specific neural network architecture to discard it and move on to a new model? Do you have a structured (generic) ...