5
$\begingroup$

Is there some type of neural network that changes the number of neurons while training?

Using this idea, the network can increase or decrease the number of neurons when the complexity of the inputs increases or decreases.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ First things first. 1.) How will a NN be aware that with it's current resources it can't solve a problem to a satisfactory level (the level is also uknown) 2.) How are you going to initialze the weights of the new neurons, since the old neurons have already done a lot of work bringing the neuron weights to a good lvl using a very small loss (by older neurons) will be tough. $\endgroup$
    – user9947
    Mar 3, 2019 at 13:16

1 Answer 1

Reset to default
4
$\begingroup$

Yes, NEAT (NeuroEvolution of Augmenting Topologies) increases the number of neurons during training. More specifically, NEAT uses evolution to introduce new neurons and connections during training, and - just as evolution - if the mutation performs poorly, gets eliminated after a few generations. This way overall performance increases over time while it keeps your network size (and computing power to run it) minimal. There's also a way to add convolution to this algorithm.

In the paper that introduced NEAT, the authors mention a completely different algorithm they tried, optimizing network hyper-parameters by choosing reasonable random values and training it multiple times. That could decrease that number as well.

Also, there is a trick to temporarily turn off specific parts of a network, which supposedly helps with overfitting.

A ReLU+backpropagation based network can "turn off" parts of the network during training, because constant 0's derivative is constant 0. In practice, you decrease the number of neurons (that is considered bad though, that's when leaky ReLU and PReLU is used instead).

Improve this answer
$\endgroup$
2
  • $\begingroup$ Does NEAT algo's change during training time? Or do they change after they see the performance (after training it completely) of a specific structure of a NN? $\endgroup$
    – user9947
    Mar 4, 2019 at 12:05
  • 1
    $\begingroup$ NEAT uses evolution to introduce new neurons and connections during training, and - just as evolution - if the mutation performs poorly, gets eliminated after a few generations. This way overall performance increases over time while it keeps your network size (and computing power to run it) minimal. There's also a way to add convolution to this algo. $\endgroup$
    – Nyos
    Mar 4, 2019 at 12:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .