Is there some type of neural network that changes the number of neurons while training?
Using this idea, the network can increase or decrease the number of neurons when the complexity of the inputs increases or decreases.
Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.Sign up to join this community
Yes, NEAT (NeuroEvolution of Augmenting Topologies) increases the number of neurons during training. More specifically, NEAT uses evolution to introduce new neurons and connections during training, and - just as evolution - if the mutation performs poorly, gets eliminated after a few generations. This way overall performance increases over time while it keeps your network size (and computing power to run it) minimal. There's also a way to add convolution to this algorithm.
In the paper that introduced NEAT, the authors mention a completely different algorithm they tried, optimizing network hyper-parameters by choosing reasonable random values and training it multiple times. That could decrease that number as well.
Also, there is a trick to temporarily turn off specific parts of a network, which supposedly helps with overfitting.
A ReLU+backpropagation based network can "turn off" parts of the network during training, because constant 0's derivative is constant 0. In practice, you decrease the number of neurons (that is considered bad though, that's when leaky ReLU and PReLU is used instead).