14
votes
Accepted
Why should the number of neurons in a hidden layer be a power of 2?
I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster.
I ...
- 30.4k
13
votes
Accepted
What exactly is a hidden state in an LSTM and RNN?
This is my own understanding of the hidden state in a recurrent network. If it's wrong, please, feel free to let me know.
Let's consider the following two input and output sequences
\begin{align}
X &...
- 1,036
12
votes
Accepted
What kind of problems require more than 2 hidden layers?
Formally, a single hidden layer is sufficient to approximate a continuous function to any desired degree of accuracy, so in that sense, you never need more than 1. This is called the Universal ...
- 7,226
7
votes
Accepted
How do I decide the optimal number of layers for a neural network?
There is a technique called Pruning in neural networks, which is used just for this same purpose.
The pruning is done on the number of hidden layers. The process ...
- 1,371
7
votes
Accepted
Is this idea to calculate the required number of hidden neurons for a single hidden layer neural network correct?
I have an idea to find the optimal number of hidden neurons required in a neural network but I'm not sure how accurate it is.
It's a complete non-starter, and there is a no such calculation possible ...
- 30.4k
7
votes
Accepted
Can the hidden layer prior to the ouput layer have less hidden units than the output layer?
A layer with bigger number of nodes than previous one is something very common. Some examples are:
strategies encoder-decoder (autoencoders) where the encoder typically has layers with a decreasing ...
- 1,283
5
votes
Why aren't there neural networks that connect the output of each layer to all next layers?
Actually, this already exists!
I happened to make a presentation of a paper that talks about this topic. These networks are called DenseNets, which stands for densely connected convolutional networks....
- 51
5
votes
Accepted
What is the purpose of the hidden layers?
"Hidden" layers really aren't all that special... a hidden layer is really no more than any layer that isn't input or output. So even a very simple 3 layer NN has 1 hidden layer. So I think the ...
- 3,757
5
votes
Accepted
What's the difference between hyperbolic tangent and sigmoid neurons?
Sigmoid > Hyperbolic tangent:
As you mentioned, the application of Sigmoid might be more convenient than hyperbolic tangent in the cases that we need a probability value at the output (as @matthew-...
5
votes
Accepted
Why is there a sigmoid function in the hidden layer of a neural network?
Let us suppose we have a network without any functions in between. Each layer consists of a linear function. i.e
layer_output = Weights.layer_input + bias
...
4
votes
Accepted
How can I understand this statement about RNNs and hidden layers?
I assume the statement was made for Elman recurrent neural networks, because as far as I know, that is the only type of neural networks for which that statement is valid.
Let's say we have an Elman ...
- 56
4
votes
Does each filter in each convolution layer create a new image?
About the images inside the CNN layers: I really recommend this article since there is no one short answer to this question and it probably will be better to experiment with it.
About the RGB input ...
- 96
4
votes
Accepted
Does each filter in each convolution layer create a new image?
You are partially correct. On CNNs the output shape per layer is defined by the amount of filters used, and the application of the filters (dilation, stride, padding, etc.).
CNNs shapes
In your ...
- 156
4
votes
Does each filter in each convolution layer create a new image?
For a 3 channel image (RGB), each filter in a convolutional layer computes a feature map which is essentially a single channel image. Typically, 2D convolutional filters are used for multichannel ...
- 156
4
votes
Why does Batch Normalization work?
I believe anything in machine learning that works, works because it flattens and smoothens the loss landscape.
Batch and layer normalization would help ensure that the feature vectors (i.e. channels) ...
- 326
3
votes
How many nodes/hidden layers are required to solve a classification problem where the boundary is a sinusoidal function?
It depends on the accuracy you want. If you had 1 neuron, it could discern things across a line, if you have 2, you could solve things across 2 lines, etc. As you increase the number of neurons, you ...
- 3,787
3
votes
How to chose dense layer size?
I am also wondering about this.
It must depend both on convolutional sub-network output size (N) and number of classes (M).
Maybe there are some rules of thumbs depending on (N, M).
Why 2 dense ...
- 31
3
votes
Accepted
How to chose dense layer size?
It's depend more on number of classes. For 20 classes 2 layers 512 should be more then enough. If you want to experiment you can try also 2 x 256 and 2 x 1024. Less then 256 may work too, but you may ...
- 705
3
votes
Accepted
How do I choose the size of the hidden state of a GRU?
Yes, your understanding of the hidden state is correct. But the size of the hidden state is a hyperparameter that needs to found by trial-and-error. There is no closed-form formula or solution which ...
- 562
3
votes
What is the purpose of the hidden layers?
Hidden layers by themselves aren't useful. If you had hidden layers that were linear, the end result would still be a linear function of the inputs, and so you could collapse an arbitrary number of ...
- 4,262
3
votes
What's the difference between hyperbolic tangent and sigmoid neurons?
I don't think it makes sense to decide activation functions based on desired properties of the output; you can easily insert a calibration step that maps the 'neural network score' to whatever units ...
- 4,262
3
votes
What exactly is a hidden state in an LSTM and RNN?
As you said, one way to look at it is definitely that the LSTM-encoder's encoding can be only understood by itself, that's why the decoder exists there. An optimisation process encoded it, why couldn'...
- 1,409
3
votes
What exactly is a hidden state in an LSTM and RNN?
I like to think of hidden states as intermediate representations of input within a neural system. The overall goal of the system is to re-represent an input in some specific way so that the system can ...
- 31
3
votes
What could be the problem when a neural network with four hidden layers with the sigmoid activation function is not learning?
Your code suggests a likely problem here: It looks like you are training a very deep neural network with sigmoidal activation functions at every layer.
The sigmoid has the property that its ...
- 9,207
3
votes
Accepted
How are non-linear surfaces formed in the training of a neural network?
Hi and welcome to the community. It's important to understand these basic concepts very clearly.
You have to first understand the basic unit of a neural network, a single node/neuron/perceptron. Let ...
- 148
2
votes
How do I decide the optimal number of layers for a neural network?
You can take a look at bayesian hyperparameter optimization as a general method of optimizing loss (or anything) as a function of the hyperparameters. But note that in general the deeper your network ...
- 2,071
2
votes
What is the purpose of the hidden layers?
Actually, the hierarchical learning explanation given by mindcrime is not that acceptable anymore (This was also indicated by Ian Goodfellow). Since there are neural networks with 150 layers or more, ...
- 304
2
votes
Accepted
What type of neural network would be most feasible for playing a realtime game?
I recommend you read up on reinforcement learning. Seeing how AirHockey is similar to the old Atari game Pong, here is a write-up (with code) about how to implement a simple neural network, that plays ...
- 136
2
votes
Are these statements about the performance of neural networks as a function of the number of hidden layers contradictory?
There are many problems requiring more than two hidden layers. Randomly select a recent Google journal paper on deep learning, you'll see their network could have something like 5 (or more) hidden ...
- 1,411
2
votes
Accepted
To what does the number of hidden layers in a neural network correspond?
This is a very interesting question that you ask. I believe, this post and this post (written by me) well address your question. However, it deserves an explanation here.
1. Fully connected networks
...
- 264
Only top scored, non community-wiki answers of a minimum length are eligible