14 votes
Accepted

Why should the number of neurons in a hidden layer be a power of 2?

I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster. I ...
user avatar
  • 23.8k
12 votes
Accepted

What kind of problems require more than 2 hidden layers?

Formally, a single hidden layer is sufficient to approximate a continuous function to any desired degree of accuracy, so in that sense, you never need more than 1. This is called the Universal ...
user avatar
7 votes
Accepted

How do I decide the optimal number of layers for a neural network?

There is a technique called Pruning in neural networks, which is used just for this same purpose. The pruning is done on the number of hidden layers. The process ...
user avatar
  • 1,351
7 votes
Accepted

Is this idea to calculate the required number of hidden neurons for a single hidden layer neural network correct?

I have an idea to find the optimal number of hidden neurons required in a neural network but I'm not sure how accurate it is. It's a complete non-starter, and there is a no such calculation possible ...
user avatar
  • 23.8k
7 votes
Accepted

What exactly is a hidden state in an LSTM and RNN?

This is my own understanding of the hidden state in a recurrent network. If it's wrong, please, feel free to let me know. Let's consider the following two input and output sequences \begin{align} X &...
user avatar
  • 946
7 votes
Accepted

Can the hidden layer prior to the ouput layer have less hidden units than the output layer?

A layer with bigger number of nodes than previous one is something very common. Some examples are: strategies encoder-decoder (autoencoders) where the encoder typically has layers with a decreasing ...
user avatar
5 votes
Accepted

What is the purpose of the hidden layers?

"Hidden" layers really aren't all that special... a hidden layer is really no more than any layer that isn't input or output. So even a very simple 3 layer NN has 1 hidden layer. So I think the ...
user avatar
  • 3,677
5 votes
Accepted

What's the difference between hyperbolic tangent and sigmoid neurons?

Sigmoid > Hyperbolic tangent: As you mentioned, the application of Sigmoid might be more convenient than hyperbolic tangent in the cases that we need a probability value at the output (as @matthew-...
user avatar
5 votes
Accepted

Why is there a sigmoid function in the hidden layer of a neural network?

Let us suppose we have a network without any functions in between. Each layer consists of a linear function. i.e layer_output = Weights.layer_input + bias ...
user avatar
4 votes

Why aren't there neural networks that connect the output of each layer to all next layers?

Actually, this already exists! I happened to make a presentation of a paper that talks about this topic. These networks are called DenseNets, which stands for densely connected convolutional networks....
user avatar
  • 41
4 votes
Accepted

How can I understand this statement about RNNs and hidden layers?

I assume the statement was made for Elman recurrent neural networks, because as far as I know, that is the only type of neural networks for which that statement is valid. Let's say we have an Elman ...
user avatar
4 votes

Does each filter in each convolution layer create a new image?

About the images inside the CNN layers: I really recommend this article since there is no one short answer to this question and it probably will be better to experiment with it. About the RGB input ...
user avatar
4 votes
Accepted

Does each filter in each convolution layer create a new image?

You are partially correct. On CNNs the output shape per layer is defined by the amount of filters used, and the application of the filters (dilation, stride, padding, etc.). CNNs shapes In your ...
user avatar
  • 156
4 votes

Does each filter in each convolution layer create a new image?

For a 3 channel image (RGB), each filter in a convolutional layer computes a feature map which is essentially a single channel image. Typically, 2D convolutional filters are used for multichannel ...
user avatar
  • 156
3 votes

How many nodes/hidden layers are required to solve a classification problem where the boundary is a sinusoidal function?

It depends on the accuracy you want. If you had 1 neuron, it could discern things across a line, if you have 2, you could solve things across 2 lines, etc. As you increase the number of neurons, you ...
user avatar
3 votes
Accepted

How to chose dense layer size?

It's depend more on number of classes. For 20 classes 2 layers 512 should be more then enough. If you want to experiment you can try also 2 x 256 and 2 x 1024. Less then 256 may work too, but you may ...
user avatar
3 votes

What is the purpose of the hidden layers?

Hidden layers by themselves aren't useful. If you had hidden layers that were linear, the end result would still be a linear function of the inputs, and so you could collapse an arbitrary number of ...
user avatar
3 votes

What's the difference between hyperbolic tangent and sigmoid neurons?

I don't think it makes sense to decide activation functions based on desired properties of the output; you can easily insert a calibration step that maps the 'neural network score' to whatever units ...
user avatar
3 votes

What could be the problem when a neural network with four hidden layers with the sigmoid activation function is not learning?

Your code suggests a likely problem here: It looks like you are training a very deep neural network with sigmoidal activation functions at every layer. The sigmoid has the property that its ...
user avatar
3 votes
Accepted

How are non-linear surfaces formed in the training of a neural network?

Hi and welcome to the community. It's important to understand these basic concepts very clearly. You have to first understand the basic unit of a neural network, a single node/neuron/perceptron. Let ...
user avatar
  • 128
2 votes

How do I decide the optimal number of layers for a neural network?

You can take a look at bayesian hyperparameter optimization as a general method of optimizing loss (or anything) as a function of the hyperparameters. But note that in general the deeper your network ...
user avatar
2 votes

What is the purpose of the hidden layers?

Actually, the hierarchical learning explanation given by mindcrime is not that acceptable anymore (This was also indicated by Ian Goodfellow). Since there are neural networks with 150 layers or more, ...
user avatar
  • 174
2 votes
Accepted

What type of neural network would be most feasible for playing a realtime game?

I recommend you read up on reinforcement learning. Seeing how AirHockey is similar to the old Atari game Pong, here is a write-up (with code) about how to implement a simple neural network, that plays ...
user avatar
2 votes

Are these statements about the performance of neural networks as a function of the number of hidden layers contradictory?

There are many problems requiring more than two hidden layers. Randomly select a recent Google journal paper on deep learning, you'll see their network could have something like 5 (or more) hidden ...
user avatar
  • 1,378
2 votes
Accepted

How do I choose the size of the hidden state of a GRU?

Yes, your understanding of the hidden state is correct. But the size of the hidden state is a hyperparameter that needs to found by trial-and-error. There is no closed-form formula or solution which ...
user avatar
  • 532
2 votes
Accepted

To what does the number of hidden layers in a neural network correspond?

This is a very interesting question that you ask. I believe, this post and this post (written by me) well address your question. However, it deserves an explanation here. 1. Fully connected networks ...
user avatar
  • 164
2 votes

How to chose dense layer size?

I am also wondering about this. It must depend both on convolutional sub-network output size (N) and number of classes (M). Maybe there are some rules of thumbs depending on (N, M). Why 2 dense ...
user avatar
2 votes
Accepted

How many hidden layers are needed for this training data set

You need to perform Hyperparameter Tuning to identify - Number of hidden layers. Number of neurons in each of the hidden layers. Dropout The activation function you use in each of your hidden layers. ...
user avatar
2 votes

Why is there a sigmoid function in the hidden layer of a neural network?

By itself, I'm not sure it's possible to know. It's possible the slides were old. Or, the intended purpose was to mention how as sigmoid ranges from 0 to 1. Mostly, it looks like it was intended to ...
user avatar
  • 146
2 votes

What exactly is a hidden state in an LSTM and RNN?

As you said, one way to look at it is definitely that the LSTM-encoder's encoding can be only understood by itself, that's why the decoder exists there. An optimisation process encoded it, why couldn'...
user avatar
  • 1,349

Only top scored, non community-wiki answers of a minimum length are eligible