Is it redunant to add more layers to a neural network with same number of neurons as the previous layer?

Question

Lets say I have a neural network with three layers and the last layer has 3 outputs. If I added additional layer of 3 neurons to the end of the network, would that be a more powerful neural network?

Here's an example picture of what I mean:

(where the circles are the neurons in the network, L1 is layer1. Also assume that all of the neurons are fully-connected to the neurons of the previous layer.)

Typically neural networks that I see the amount of neurons per network typically decrease as there are layers. But is it theoretically possible to a really powerful network by signiciantly increasing the depth of the network?

For example consider the following case:

Does stacking extra layers on the end of a neural network allow it to learn more complex tasks? My guess would be that stacking extra layers only helps if you are distilling the information down to less neurons per layer. But with more advanced neural networks like GPT-3, what has been said to me is what is important is the total amount of tunable parameters, and that the particular structure or depth of the network isn't important (when holding the total number of tunable parameters constant).

So phrasing this another way: If, for example, I wanted to create a neural network at the scale of a GPT-3 level, (ignoring the intricacies of the transformer architecture and just assuming that it is a basic neural network), is it possible that instead of adding additional neurons per layer, I can simply add more layers of the same neuron size to scale-up the total weight parameters?

Answer 1

A deeper network will have more capacity, regardless of whether new layers have the same number of neurons as the previous layers, fewer, or more.

More capacity means that in theory the network can learn more complex functions. It does not mean that the network will then be better on any given task.

One important thing to note: This is only the case if you have a non linear activation function. If you do not have a non linear activation function between layers, then you are correct that each added layer, despite adding more parameters, does not add capacity to the network.

For further reference, I recommend reading this chapter of the deep learning book: https://www.deeplearningbook.org/contents/mlp.html

which contains all the information above (perhaps with more technical jargon) and more.

For a greater understanding of how adding layers affects capacity, this article will be useful: https://en.wikipedia.org/wiki/Universal_approximation_theorem