12
$\begingroup$

These types of questions may be problem-dependent, but I have tried to find research that addresses the question whether the number of hidden layers and their size (number of neurons in each layer) really matter or not.

So my question is, does it really matter if we for example have 1 large hidden layer of 1000 neurons vs. 10 hidden layers with 100 neurons each?

$\endgroup$
11
$\begingroup$

Basically, having multiple layers (aka a deep network) makes your network more eager to recognize certain aspects of input data. For example, if you have the details of a house (size, lawn size, location etc.) as input and want to predict the price. The first layer may predict:

  • Big area, higher price
  • Small amount of bedrooms, lower price

The second layer might conclude:

  • Big area + small amount of bedrooms = large bedrooms = +- effect

Yes, one layer can also 'detect' the stats, however it will require more neurons as it cannot rely on other neurons to do 'parts' of the total calculation required to detect that stat.

Check out this answer

$\endgroup$
  • $\begingroup$ Thank you so much for your answer. Just to clarify, when you write "it makes your network [...]", are you referring to the case when I have many hidden layers with fewer neurons each rather than having more neurons in fewer layers? $\endgroup$ – Stephen Johnson May 4 '17 at 19:31
  • $\begingroup$ @StephenJohnson oops, I edited the question. I'm referring to the deep network (multiple layers). $\endgroup$ – Thomas W May 4 '17 at 19:36
  • $\begingroup$ Nice answer, thanks again. Maybe I should continue this in another thread, but do you think the same kind of reasoning applies to recurrent neural networks such as GRU or LSTM? $\endgroup$ – Stephen Johnson May 4 '17 at 20:18
  • $\begingroup$ @StephenJohnson do you mean one layered recurrent networks vs multilayered recurrent networks or do you mean because of their recurrent connections? $\endgroup$ – Thomas W May 4 '17 at 21:20
  • $\begingroup$ I mean generally, because of they have recurrent connections allowing them to map contexts over longer distances, do such networks benefit from being Deep in the same way a regular feedforward network would? Maybe they can't be compared like that since recurrent networks typically are used when sequential data, such as audio, is used. $\endgroup$ – Stephen Johnson May 4 '17 at 22:22
4
$\begingroup$

There are so many aspects.

1. Training: Training deep nets is a hard job due to the vanishing (rearly exploding) gradient problem. So building a 10x100 neural-net is not recommended.

2. Trained network performance:

  • Information loss: The classical usage of neural nets is the classification problem. Which means we want to get some well defined information from the data. (Ex. Is there a face in the picture or not.) So usually classification problem has a lot of input, and few output, whats more the size of the hidden layers are descend from input to output. However, we loss information using less neurons layer by layer. (Ie. We cannot reproduce the original image based on the fact that is there a face on it or no.) So you must know that you loss information using 100 neurons if the size of the input is (lets say) 1000.
  • Information complexity: However the deeper nets (as Tomas W mentioned) can fetch more complex information from the input data. Inspite of this its not recommended to use 10 fully connected layers. Its recommended to use convolutional/relu/maxpooling or other type of layers. Firest layers can compress the some essential part of the inputs. (Ex is there any line in a specific part of the picture) Second layers can say: There is a specific shape in this place in the picture. Etc etc.

So deeper nets are more "clever" but 10x100 net structure is a good choice.

$\endgroup$
1
$\begingroup$

If the problem you are solving is linearly separable, one layer of 1000 neurons can do better job than 10 layers with each of 100 neurons. If the problem is non linear and not convex, then you need deep neural nets.

$\endgroup$
  • 2
    $\begingroup$ If the problem is linearly separable you don't need hidden layers at all $\endgroup$ – Amedeo Baragiola Dec 24 '18 at 14:55
0
$\begingroup$

I think you have a confusion in the basics of the neural networks. Every layer has a separate activation function and input/output connection weights.

The output of the first hidden layer will be multiplied by a weight, processed by an activation function in the next layer and so on. Single layer neural networks are very limited for simple tasks, deeper NN can perform far better than a single layer.

However, do not use more than layer if your application is not fairly complex. In conclusion, 100 neurons layer does not mean better neural network than 10 layers x 10 neurons but 10 layers are something imaginary unless you are doing deep learning. start with 10 neurons in the hidden layer and try to add layers or add more neurons to the same layer to see the difference. learning with more layers will be easier but more training time is required.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.