0
$\begingroup$

Just came across this article on GPT-3, and that lead me to the question:

In order to make a certain kind of neural network architecture smarter all one needs to do is to make it bigger?

Also, if that is true, how does the importance of computer power relates with the importance of fine-tuning/algorithmic improvement?

$\endgroup$
  • $\begingroup$ Maybe you should summarise what the article says (for those not interested in reading it) that made you think that a bigger architecture leads to more "intelligence". The thing that increases if the number of parameters of the model increases is the capacity (i.e. roughly, the number of functions that the neural network can represent). So, what's the definition of intelligence you are using here? $\endgroup$ – nbro Jul 12 at 12:57
2
$\begingroup$

First of all, there is no real 'intelligence' innate to artificial Neural Networks (NNs). All they do is trying to approximate a mathematical function with a certain degree of generalization (hopefully without learning a given dataset by heart, i.e. hopefully without overfitting).

The more nodes (or neurons) you include into the network, the more complex a function can be that a network can learn to approximate. It's similar to high-school math: The higher the degree of some polynomial, the better the polynomial can be adjusted to fit some observation to be modeled; with the only difference being that NNs commonly include non-linearities and are trained via some kind of stochastic gradient descent.

So, yes. The more nodes a model possesses, the higher the so-called model capacity, i.e. the higher the degree of freedom a NN-model has to fit some function. After all, NN are said to be universal function approximators - given they have enough internal nodes in their hidden layer(s) to fit some given function.

In practice, however, you don't want to blow up a model architecture unnecessarily, since this commonly results in overfitting if it doesn't cause some instabilities of the training procedure instead.

Generally, the larger the model to be trained, the higher the computational cost to train the network.

A common suggestion is to reduce the number of nodes in a network at the expense of increasing a network's depth, i.e. the number of hidden layers. Often, that can help reduce the demand for excessively many nodes.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.