Lets say I have a neural network with three layers and the last layer has 3 outputs. If I added additional layer of 3 neurons to the end of the network, would that be a more powerful neural network?

Here's an example picture of what I mean: enter image description here

(where the circles are the neurons in the network, L1 is layer1. Also assume that all of the neurons are fully-connected to the neurons of the previous layer.)

Typically neural networks that I see the amount of neurons per network typically decrease as there are layers. But is it theoretically possible to a really powerful network by signiciantly increasing the depth of the network?

For example consider the following case:

enter image description here Does stacking extra layers on the end of a neural network allow it to learn more complex tasks? My guess would be that stacking extra layers only helps if you are distilling the information down to less neurons per layer. But with more advanced neural networks like GPT-3, what has been said to me is what is important is the total amount of tunable parameters, and that the particular structure or depth of the network isn't important (when holding the total number of tunable parameters constant).

So phrasing this another way: If, for example, I wanted to create a neural network at the scale of a GPT-3 level, (ignoring the intricacies of the transformer architecture and just assuming that it is a basic neural network), is it possible that instead of adding additional neurons per layer, I can simply add more layers of the same neuron size to scale-up the total weight parameters?

  • $\begingroup$ Think again. Where does "the total amount of tunable parameters" come from? From "the amount of nodes or layers". It does not matter how big or small the network is. $\endgroup$
    – lpounng
    Commented Nov 23, 2022 at 2:54
  • $\begingroup$ @lpounng, sorry but I dont understand what you mean here. Of course a larger network (one with more parameters) is neccessary for more complicated problems -- so I don't follow. $\endgroup$ Commented Nov 23, 2022 at 14:40
  • $\begingroup$ Referring to this line, "...what is important is the total amount of tunable parameters, and that the amount of nodes or layers is not as important". $\endgroup$
    – lpounng
    Commented Nov 28, 2022 at 8:17
  • $\begingroup$ I know what line you are referring to, but don't understand your question. $\endgroup$ Commented Nov 29, 2022 at 15:25
  • $\begingroup$ It was not a question, but a statement. "the total amount of tunable parameters" is proportional to "the amount of nodes or layers", regardless of the scale of model. So the statement "what is important is the total amount of tunable parameters, and that the amount of nodes or layers is not as important" does not make sense. $\endgroup$
    – lpounng
    Commented Nov 30, 2022 at 1:23

1 Answer 1


A deeper network will have more capacity, regardless of whether new layers have the same number of neurons as the previous layers, fewer, or more.

More capacity means that in theory the network can learn more complex functions. It does not mean that the network will then be better on any given task.

One important thing to note: This is only the case if you have a non linear activation function. If you do not have a non linear activation function between layers, then you are correct that each added layer, despite adding more parameters, does not add capacity to the network.

For further reference, I recommend reading this chapter of the deep learning book: https://www.deeplearningbook.org/contents/mlp.html

which contains all the information above (perhaps with more technical jargon) and more.

For a greater understanding of how adding layers affects capacity, this article will be useful: https://en.wikipedia.org/wiki/Universal_approximation_theorem

  • $\begingroup$ "If you do not have a non linear activation function" - is that simply saying you have a linear activation function? 😊 $\endgroup$
    – eddiewould
    Commented Dec 9, 2022 at 3:23
  • 1
    $\begingroup$ @eddiewould Nice reversal of the double negative :) But it also could mean that you lack an activation function entirely. $\endgroup$ Commented Dec 9, 2022 at 4:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .