Skip to main content
Share Your Experience: Take the 2024 Developer Survey
14 votes
Accepted

Why should the number of neurons in a hidden layer be a power of 2?

I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster. I ...
Neil Slater's user avatar
  • 32.4k
14 votes
Accepted

What exactly is a hidden state in an LSTM and RNN?

This is my own understanding of the hidden state in a recurrent network. If it's wrong, please, feel free to let me know. Let's consider the following two input and output sequences \begin{align} X &...
Eka's user avatar
  • 1,066
12 votes
Accepted

What kind of problems require more than 2 hidden layers?

Formally, a single hidden layer is sufficient to approximate a continuous function to any desired degree of accuracy, so in that sense, you never need more than 1. This is called the Universal ...
NietzscheanAI's user avatar
7 votes
Accepted

How do I decide the optimal number of layers for a neural network?

There is a technique called Pruning in neural networks, which is used just for this same purpose. The pruning is done on the number of hidden layers. The process ...
Dawny33's user avatar
  • 1,371
7 votes
Accepted

Is this idea to calculate the required number of hidden neurons for a single hidden layer neural network correct?

I have an idea to find the optimal number of hidden neurons required in a neural network but I'm not sure how accurate it is. It's a complete non-starter, and there is a no such calculation possible ...
Neil Slater's user avatar
  • 32.4k
7 votes
Accepted

Can the hidden layer prior to the ouput layer have less hidden units than the output layer?

A layer with bigger number of nodes than previous one is something very common. Some examples are: strategies encoder-decoder (autoencoders) where the encoder typically has layers with a decreasing ...
pasaba por aqui's user avatar
5 votes

Why aren't there neural networks that connect the output of each layer to all next layers?

Actually, this already exists! I happened to make a presentation of a paper that talks about this topic. These networks are called DenseNets, which stands for densely connected convolutional networks....
Armando's user avatar
  • 51
5 votes
Accepted

What is the purpose of the hidden layers?

"Hidden" layers really aren't all that special... a hidden layer is really no more than any layer that isn't input or output. So even a very simple 3 layer NN has 1 hidden layer. So I think the ...
mindcrime's user avatar
  • 3,757
5 votes
Accepted

What's the difference between hyperbolic tangent and sigmoid neurons?

Sigmoid > Hyperbolic tangent: As you mentioned, the application of Sigmoid might be more convenient than hyperbolic tangent in the cases that we need a probability value at the output (as @matthew-...
Borhan Kazimipour's user avatar
5 votes
Accepted

Why is there a sigmoid function in the hidden layer of a neural network?

Let us suppose we have a network without any functions in between. Each layer consists of a linear function. i.e layer_output = Weights.layer_input + bias ...
Sooryakiran Pallikulathil's user avatar
4 votes
Accepted

How can I understand this statement about RNNs and hidden layers?

I assume the statement was made for Elman recurrent neural networks, because as far as I know, that is the only type of neural networks for which that statement is valid. Let's say we have an Elman ...
sander2798's user avatar
4 votes

Does each filter in each convolution layer create a new image?

About the images inside the CNN layers: I really recommend this article since there is no one short answer to this question and it probably will be better to experiment with it. About the RGB input ...
Sahar Sela's user avatar
4 votes
Accepted

Does each filter in each convolution layer create a new image?

You are partially correct. On CNNs the output shape per layer is defined by the amount of filters used, and the application of the filters (dilation, stride, padding, etc.). CNNs shapes In your ...
adn's user avatar
  • 156
4 votes

Does each filter in each convolution layer create a new image?

For a 3 channel image (RGB), each filter in a convolutional layer computes a feature map which is essentially a single channel image. Typically, 2D convolutional filters are used for multichannel ...
tynowell's user avatar
  • 156
4 votes

Why does Batch Normalization work?

I believe anything in machine learning that works, works because it flattens and smoothens the loss landscape. Batch and layer normalization would help ensure that the feature vectors (i.e. channels) ...
Tom Huntington's user avatar
3 votes

How many nodes/hidden layers are required to solve a classification problem where the boundary is a sinusoidal function?

It depends on the accuracy you want. If you had 1 neuron, it could discern things across a line, if you have 2, you could solve things across 2 lines, etc. As you increase the number of neurons, you ...
Jaden Travnik's user avatar
3 votes

How to chose dense layer size?

I am also wondering about this. It must depend both on convolutional sub-network output size (N) and number of classes (M). Maybe there are some rules of thumbs depending on (N, M). Why 2 dense ...
lostdatum's user avatar
3 votes
Accepted

How to chose dense layer size?

It's depend more on number of classes. For 20 classes 2 layers 512 should be more then enough. If you want to experiment you can try also 2 x 256 and 2 x 1024. Less then 256 may work too, but you may ...
mirror2image's user avatar
3 votes
Accepted

How do I choose the size of the hidden state of a GRU?

Yes, your understanding of the hidden state is correct. But the size of the hidden state is a hyperparameter that needs to found by trial-and-error. There is no closed-form formula or solution which ...
varsh's user avatar
  • 562
3 votes

What is the purpose of the hidden layers?

Hidden layers by themselves aren't useful. If you had hidden layers that were linear, the end result would still be a linear function of the inputs, and so you could collapse an arbitrary number of ...
Matthew Gray's user avatar
  • 4,262
3 votes

What's the difference between hyperbolic tangent and sigmoid neurons?

I don't think it makes sense to decide activation functions based on desired properties of the output; you can easily insert a calibration step that maps the 'neural network score' to whatever units ...
Matthew Gray's user avatar
  • 4,262
3 votes

What exactly is a hidden state in an LSTM and RNN?

As you said, one way to look at it is definitely that the LSTM-encoder's encoding can be only understood by itself, that's why the decoder exists there. An optimisation process encoded it, why couldn'...
ashenoy's user avatar
  • 1,409
3 votes

What exactly is a hidden state in an LSTM and RNN?

I like to think of hidden states as intermediate representations of input within a neural system. The overall goal of the system is to re-represent an input in some specific way so that the system can ...
ticiarai's user avatar
3 votes

What exactly is a hidden state in an LSTM and RNN?

The hidden state in a RNN is basically just like a hidden layer in a regular feed-forward network - it just happens to also be used as an additional input to the RNN at the next time step. A simple ...
Burrito's user avatar
  • 141
3 votes

What could be the problem when a neural network with four hidden layers with the sigmoid activation function is not learning?

Your code suggests a likely problem here: It looks like you are training a very deep neural network with sigmoidal activation functions at every layer. The sigmoid has the property that its ...
John Doucette's user avatar
3 votes
Accepted

How are non-linear surfaces formed in the training of a neural network?

Hi and welcome to the community. It's important to understand these basic concepts very clearly. You have to first understand the basic unit of a neural network, a single node/neuron/perceptron. Let ...
Ananda's user avatar
  • 148
2 votes

How do I decide the optimal number of layers for a neural network?

You can take a look at bayesian hyperparameter optimization as a general method of optimizing loss (or anything) as a function of the hyperparameters. But note that in general the deeper your network ...
k.c. sayz 'k.c sayz''s user avatar
2 votes

What is the purpose of the hidden layers?

Actually, the hierarchical learning explanation given by mindcrime is not that acceptable anymore (This was also indicated by Ian Goodfellow). Since there are neural networks with 150 layers or more, ...
Enes's user avatar
  • 324
2 votes
Accepted

What type of neural network would be most feasible for playing a realtime game?

I recommend you read up on reinforcement learning. Seeing how AirHockey is similar to the old Atari game Pong, here is a write-up (with code) about how to implement a simple neural network, that plays ...
Žiga Sajovic's user avatar
2 votes

Are these statements about the performance of neural networks as a function of the number of hidden layers contradictory?

There are many problems requiring more than two hidden layers. Randomly select a recent Google journal paper on deep learning, you'll see their network could have something like 5 (or more) hidden ...
SmallChess's user avatar
  • 1,411

Only top scored, non community-wiki answers of a minimum length are eligible