4
$\begingroup$

Is there some type of neural network that changes the number of neurons while training?

Using this idea, the network can increase or decrease the number of neurons when the complexity of the inputs increases or decreases.

$\endgroup$
  • 1
    $\begingroup$ First things first. 1.) How will a NN be aware that with it's current resources it can't solve a problem to a satisfactory level (the level is also uknown) 2.) How are you going to initialze the weights of the new neurons, since the old neurons have already done a lot of work bringing the neuron weights to a good lvl using a very small loss (by older neurons) will be tough. $\endgroup$ – DuttaA Mar 3 at 13:16
3
$\begingroup$

Yes, NEAT (NeuroEvolution of Augmenting Topologies) increases the number of neurons during training. More specifically, NEAT uses evolution to introduce new neurons and connections during training, and - just as evolution - if the mutation performs poorly, gets eliminated after a few generations. This way overall performance increases over time while it keeps your network size (and computing power to run it) minimal. There's also a way to add convolution to this algorithm.

In the paper that introduced NEAT, the authors mention a completely different algorithm they tried, optimizing network hyper-parameters by choosing reasonable random values and training it multiple times. That could decrease that number as well.

Also, there is a trick to temporarily turn off specific parts of a network, which supposedly helps with overfitting.

A ReLU+backpropagation based network can "turn off" parts of the network during training, because constant 0's derivative is constant 0. In practice, you decrease the number of neurons (that is considered bad though, that's when leaky ReLU and PReLU is used instead).

$\endgroup$
  • $\begingroup$ Does NEAT algo's change during training time? Or do they change after they see the performance (after training it completely) of a specific structure of a NN? $\endgroup$ – DuttaA Mar 4 at 12:05
  • 1
    $\begingroup$ NEAT uses evolution to introduce new neurons and connections during training, and - just as evolution - if the mutation performs poorly, gets eliminated after a few generations. This way overall performance increases over time while it keeps your network size (and computing power to run it) minimal. There's also a way to add convolution to this algo. $\endgroup$ – Nyos Mar 4 at 12:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.