All Questions
Tagged with hyper-parameters neural-networks
31 questions
3
votes
1
answer
182
views
Can a ML model learn the hyperparameters landscape?
(I assume that this is not possible because I've never seen anyone talk about this.)
Let's take a classic MLP (named f) that, for example classify some images (from ...
1
vote
0
answers
18
views
Looking for a way to train a model to learn optimal parameters/hyperparameters of clustering
I have 5000 docs, each is a review. For each review, i'm plotting the sentences in a semantic dimension. Now, I'm applying clustering to these points for each review. The success of my model depends ...
- 11
0
votes
1
answer
38
views
Patterns binary classification - model doesn't overfit
I am working on a very basic binary classification problem. For each set of four float numbers $(x,y,z,w)$, I want to check if they fall or not into one category.
I have written a model with 3 dense ...
- 123
0
votes
1
answer
129
views
How to approach a toy classification problem using a neural network?
The toy problem:
50 unique numbers are randomly selected from number 0 to 99.
If number 1 appears in the selection but number 2 doesn't, the selection is labelled as "1".
If number 2 ...
- 1
0
votes
1
answer
226
views
Do different architectures really make a difference or is it just a matter of the training process?
I was wondering which influence different architectures for deep learning truly have on the performance. Of course, substantial changes in the paradigms we use when building neural networks (such as ...
- 121
1
vote
1
answer
130
views
Are there any guidelines on picking hyperparameters for Deep Reinforcement Learning?
I am trying to learn machine learning from Andrew NG's Machine learning specialization on Coursera. In the chapter about reinforcement learning Andrew NG said that if you do not select correct ...
1
vote
2
answers
1k
views
What should I do if my validation score is good, but my test score is bad?
I've trained my artificial neural network, and, as per standard practice, I've picked out the one neural network throughout training that did the best on my validation dataset. That is, the neural ...
- 113
0
votes
1
answer
270
views
Why should data batches in a neural network have an equal size?
Why should data batches in a neural network have an equal size?
I have seen some recent research works on making the batch size dynamic, but still, I can't find an answer to my question.
- 123
0
votes
3
answers
587
views
What can I infer if my model is converging extremely fast?
I am running a model with fixed hyperparameters. To my surprise/shock, the model converged extremely fast with the least loss possible.
I want to know the causes of this phenomenon. I have the ...
- 3,990
1
vote
2
answers
2k
views
Is there any relationship between the batch size and the number of epochs?
I am currently running a program with a batch size of 17 instead of batch size 32. The benchmark results are obtained at a batch size of 32 with the number of epochs 700.
Now I am running with batch ...
- 3,990
3
votes
1
answer
579
views
Why is the input layer of a neural network usually not counted?
I came across the following statement from the caption of figure 7.8 from the textbook Neural Networks and Neural Language Models
the input layer is usually not counted when enumerating layers
Why ...
- 3,990
3
votes
1
answer
116
views
What is the most statistically acceptable method for tuning neural network hyperparameters on very small datasets?
Neural networks are usually evaluated by dividing a dataset into three splits:
training,
validation, and
test
The idea is that critical hyperparameters of the network such as the number of epochs ...
- 411
1
vote
0
answers
238
views
How does noise input size affect fake image generation with GANs?
In Generative Adversarial Networks, the Generator takes noise vector as input and feeds it forward to create an image. The noise vector consists of random numbers sampled from the normal distribution. ...
- 793
1
vote
1
answer
464
views
Should the range and initial values of weights and biases be adjusted to fit input and output data?
As a routine (in typical everyday tasks) of a data scientist, should they usually decide about weights and biases range and initial values as a function of which data they are planning to insert as ...
- 303
1
vote
1
answer
124
views
Why one unit in the layers of neural network is not enough?
In a deep connected network, when every unit gets all the input features(X) so it has one parameter for every feature and every unit tweaks its parameters for loss optimization. What if we use only ...
- 111
3
votes
1
answer
139
views
How are training hyperparameters determined for large models?
When training a relatively small DL model, which takes several hours to train, I typically start with some starting points from literature and then use a trial-and-error or grid-search approach to ...
- 133
3
votes
1
answer
616
views
Why does every neuron in hidden layers of a multi-layer perceptron typically have the same activation function? [duplicate]
Why does every neuron in a hidden layer of a multi-layer perceptron (MLP) typically have the same activation function as every other neuron in the same or other hidden layers (so I exclude the output ...
- 805
1
vote
1
answer
351
views
How to know if the hyperparameters of a neural network relate to each other?
According this thread some hyperparameters are independent from each other while some are directly related.
One of the answers give an example where two hyperparameters affect each other.
For ...
- 236
1
vote
0
answers
125
views
Why is the number of neurons used in various neural networks power of 2?
I have noticed that almost all tutorials take the number of neurons as a power of 2. Is there any proper mathematical and well-proven reason for that?
If you sometimes change it to some other odd ...
- 164
3
votes
0
answers
41
views
Which hyper-parameters are considered in neural architecture search?
I want to understand automatic Neural Architecture Search (NAS). I read already multiple papers, but I cannot figure out what the actual search space of NAS is / how are classical hyper-parameters ...
- 73
4
votes
1
answer
137
views
When using Neural Architecture Search, how are the hyper-parameters chosen?
I have read a lot about NAS, but I still do not understand one concept: When setting up a neural network, hyperparameters (such as the learning rate, dropout rate, batch size, filter size, etc.) need ...
- 73
6
votes
1
answer
2k
views
Is this idea to calculate the required number of hidden neurons for a single hidden layer neural network correct?
I have an idea to find the optimal number of hidden neurons required in a neural network, but I'm not sure how accurate it is.
Assuming that it has only 1 hidden layer, it is a classification problem ...
- 205
5
votes
2
answers
270
views
In a neural network, by how much does the number of neurons typically vary from layer to layer?
In a neural network, by how much does the number of neurons typically vary from layer to layer?
Note that I am NOT asking how to find the optimal number of neurons per layer.
As a hardware design ...
5
votes
1
answer
92
views
How do you efficiently choose the hyper-parameters of a neural network?
How do you efficiently choose the hyper-parameters of a neural network (e.g. the learning rate, number of layer, weights, etc.)?
- 51
6
votes
1
answer
2k
views
How should we choose the dimensions of the encoding layer in auto-encoders?
How should we choose the dimensions of the encoding layer in auto-encoders?
- 101
2
votes
1
answer
669
views
Maximum number of neurons in a layer given number of neurons in previous layer
Consider an extremely complicated feed-forward neural network training example but with no need of computational efficiency or limiting of processing time.
What is the maximum number of hidden ...
- 165
24
votes
3
answers
13k
views
How to choose an activation function for the hidden layers?
I choose the activation function for the output layer depending on the output that I need and the properties of the activation function that I know. For example, I choose the sigmoid function when I'm ...
- 706
3
votes
3
answers
2k
views
Why must the momentum factor be in the range 0-1?
Why is it a bad idea to have a momentum factor greater than 1? What are the mathematical motivations/reasons?
- 61
2
votes
1
answer
689
views
How to design a neural network to predict the quadrant where a given point lies?
I've written a single perceptron that can predict whether a point is above or below a straight-line graph, given the correct training data and using a sign activation function.
Now, I'm trying to ...
- 205
5
votes
1
answer
372
views
How do I design a neural network that breaks a 5-letter word into its corresponding syllables?
I am going to design a neural network which will be able to break a 5-letter word into its corresponding syllables (hybrid syllables, I mean it will not strictly adhere to grammatical syllable rules ...
- 174
33
votes
4
answers
2k
views
How to find the optimal number of neurons per layer?
When you're writing your algorithm, how do you know how many neurons you need per single layer? Are there any methods for finding the optimal number of them, or is it a rule of thumb?
- 10.5k