Lets say I have a neural network with three layers and the last layer has 3 outputs. If I added additional layer of 3 neurons to the end of the network, would that be a more powerful neural network?
(where the circles are the neurons in the network, L1 is layer1. Also assume that all of the neurons are fully-connected to the neurons of the previous layer.)
Typically neural networks that I see the amount of neurons per network typically decrease as there are layers. But is it theoretically possible to a really powerful network by signiciantly increasing the depth of the network?
For example consider the following case:
Does stacking extra layers on the end of a neural network allow it to learn more complex tasks? My guess would be that stacking extra layers only helps if you are distilling the information down to less neurons per layer. But with more advanced neural networks like GPT-3, what has been said to me is what is important is the total amount of tunable parameters, and that the particular structure or depth of the network isn't important (when holding the total number of tunable parameters constant).
So phrasing this another way: If, for example, I wanted to create a neural network at the scale of a GPT-3 level, (ignoring the intricacies of the transformer architecture and just assuming that it is a basic neural network), is it possible that instead of adding additional neurons per layer, I can simply add more layers of the same neuron size to scale-up the total weight parameters?