14
votes
Accepted
Why should the number of neurons in a hidden layer be a power of 2?
I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster.
I ...
- 33.3k
14
votes
Accepted
What exactly is a hidden state in an LSTM and RNN?
This is my own understanding of the hidden state in a recurrent network. If it's wrong, please, feel free to let me know.
Let's consider the following two input and output sequences
\begin{align}
X &...
- 1,096
7
votes
Accepted
Is this idea to calculate the required number of hidden neurons for a single hidden layer neural network correct?
I have an idea to find the optimal number of hidden neurons required in a neural network but I'm not sure how accurate it is.
It's a complete non-starter, and there is a no such calculation possible ...
- 33.3k
7
votes
Accepted
Can the hidden layer prior to the ouput layer have less hidden units than the output layer?
A layer with bigger number of nodes than previous one is something very common. Some examples are:
strategies encoder-decoder (autoencoders) where the encoder typically has layers with a decreasing ...
- 1,293
5
votes
Accepted
What's the difference between hyperbolic tangent and sigmoid neurons?
Sigmoid > Hyperbolic tangent:
As you mentioned, the application of Sigmoid might be more convenient than hyperbolic tangent in the cases that we need a probability value at the output (as @matthew-...
5
votes
Why aren't there neural networks that connect the output of each layer to all next layers?
Actually, this already exists!
I happened to make a presentation of a paper that talks about this topic. These networks are called DenseNets, which stands for densely connected convolutional networks....
- 51
5
votes
Accepted
Why is there a sigmoid function in the hidden layer of a neural network?
Let us suppose we have a network without any functions in between. Each layer consists of a linear function. i.e
layer_output = Weights.layer_input + bias
...
5
votes
What exactly is a hidden state in an LSTM and RNN?
The hidden state in a RNN is basically just like a hidden layer in a regular feed-forward network - it just happens to also be used as an additional input to the RNN at the next time step.
A simple ...
- 161
4
votes
Does each filter in each convolution layer create a new image?
About the images inside the CNN layers: I really recommend this article since there is no one short answer to this question and it probably will be better to experiment with it.
About the RGB input ...
- 96
4
votes
Accepted
Does each filter in each convolution layer create a new image?
You are partially correct. On CNNs the output shape per layer is defined by the amount of filters used, and the application of the filters (dilation, stride, padding, etc.).
CNNs shapes
In your ...
- 156
4
votes
Does each filter in each convolution layer create a new image?
For a 3 channel image (RGB), each filter in a convolutional layer computes a feature map which is essentially a single channel image. Typically, 2D convolutional filters are used for multichannel ...
- 156
4
votes
Why does Batch Normalization work?
I believe anything in machine learning that works, works because it flattens and smoothens the loss landscape.
Batch and layer normalization would help ensure that the feature vectors (i.e. channels) ...
- 366
3
votes
What exactly is a hidden state in an LSTM and RNN?
As you said, one way to look at it is definitely that the LSTM-encoder's encoding can be only understood by itself, that's why the decoder exists there. An optimisation process encoded it, why couldn'...
- 1,419
3
votes
Accepted
How do I choose the size of the hidden state of a GRU?
Yes, your understanding of the hidden state is correct. But the size of the hidden state is a hyperparameter that needs to found by trial-and-error. There is no closed-form formula or solution which ...
- 562
3
votes
How to chose dense layer size?
I am also wondering about this.
It must depend both on convolutional sub-network output size (N) and number of classes (M).
Maybe there are some rules of thumbs depending on (N, M).
Why 2 dense ...
- 31
3
votes
Accepted
How to chose dense layer size?
It's depend more on number of classes. For 20 classes 2 layers 512 should be more then enough. If you want to experiment you can try also 2 x 256 and 2 x 1024. Less then 256 may work too, but you may ...
- 735
3
votes
How many nodes/hidden layers are required to solve a classification problem where the boundary is a sinusoidal function?
It depends on the accuracy you want. If you had 1 neuron, it could discern things across a line, if you have 2, you could solve things across 2 lines, etc. As you increase the number of neurons, you ...
- 3,857
3
votes
What exactly is a hidden state in an LSTM and RNN?
I like to think of hidden states as intermediate representations of input within a neural system. The overall goal of the system is to re-represent an input in some specific way so that the system can ...
- 31
3
votes
What could be the problem when a neural network with four hidden layers with the sigmoid activation function is not learning?
Your code suggests a likely problem here: It looks like you are training a very deep neural network with sigmoidal activation functions at every layer.
The sigmoid has the property that its ...
- 9,392
3
votes
Accepted
How are non-linear surfaces formed in the training of a neural network?
Hi and welcome to the community. It's important to understand these basic concepts very clearly.
You have to first understand the basic unit of a neural network, a single node/neuron/perceptron. Let ...
- 158
2
votes
How do I decide the optimal number of layers for a neural network?
You can take a look at bayesian hyperparameter optimization as a general method of optimizing loss (or anything) as a function of the hyperparameters. But note that in general the deeper your network ...
- 2,111
2
votes
What is the purpose of the hidden layers?
Actually, the hierarchical learning explanation given by mindcrime is not that acceptable anymore (This was also indicated by Ian Goodfellow). Since there are neural networks with 150 layers or more, ...
- 324
2
votes
Accepted
What type of neural network would be most feasible for playing a realtime game?
I recommend you read up on reinforcement learning. Seeing how AirHockey is similar to the old Atari game Pong, here is a write-up (with code) about how to implement a simple neural network, that plays ...
- 136
2
votes
Are these statements about the performance of neural networks as a function of the number of hidden layers contradictory?
There are many problems requiring more than two hidden layers. Randomly select a recent Google journal paper on deep learning, you'll see their network could have something like 5 (or more) hidden ...
- 1,421
2
votes
Accepted
To what does the number of hidden layers in a neural network correspond?
This is a very interesting question that you ask. I believe, this post and this post (written by me) well address your question. However, it deserves an explanation here.
1. Fully connected networks
...
- 284
2
votes
Accepted
How many hidden layers are needed for this training data set
You need to perform Hyperparameter Tuning to identify -
Number of hidden layers.
Number of neurons in each of the hidden layers.
Dropout
The activation function you use in each of your hidden layers.
...
- 158
2
votes
Why is there a sigmoid function in the hidden layer of a neural network?
By itself, I'm not sure it's possible to know. It's possible the slides were old. Or, the intended purpose was to mention how as sigmoid ranges from 0 to 1. Mostly, it looks like it was intended to ...
- 146
2
votes
Accepted
Do all neurons in a layer have the same activation function?
From here:
Using other activation functions don’t provide significant improvement in performance and tweaking them doesn’t provide any big improvement. So as per simplicity we use same activation ...
- 1,406
2
votes
Accepted
Is it expected that adding an additional hidden layer to my 3-layer ANN reduces accuracy significantly?
You probably got the back propagation wrong. I have done a test on the accuracy on adding an extra layer and the accuracy went up from 94% to 96% for me. See this for details:
https://colab.research....
- 1,755
2
votes
Accepted
Are there deep neural networks that have inputs connected with deeper hidden layers?
This type of connections are called skip or residual connections. There are numerous works which employs this type of mechanism, for example: ResNet, SkipRNN. In addition here you can find a paper ...
- 1,158
Only top scored, non community-wiki answers of a minimum length are eligible