For an experiment that I'm working on, I want to train a deep network in a special way. I want to initialize and train a small network first, then, in a specific way, I want to increase network depth leading to a bigger network which is subsequently to be trained. This process will be repeated until one reaches the desired depth. It would be great if anybody heard of anything similar and could point out to me some related work. I think in some paper I read something about a related technique where people used something similar but I don't find it anymore.
4
$\begingroup$
$\endgroup$
-
$\begingroup$ it is common when dealing with images to train this way using progressive resizing $\endgroup$ – juvian Aug 13 at 15:55
-
$\begingroup$ @juvian thanks. that keyword was helpful $\endgroup$ – Leroy Od Aug 18 at 15:56
5
$\begingroup$
$\endgroup$
Neuroevolution Through Augmenting Topologies or NEAT may be what you are referring to. The original paper by Kenneth O. Stanley is here
NEAT combines a neural network and a genetic algorithm. Instead of using back propagation or gradient descent to "train" your network, NEAT creates a population of very simple neural networks (no connections) and evolves them with fitness evaluation, crossover, and mutation. The genome syntax: every connection gene has a few settings. In node, Out node, Weight of connection, activated, and innovation. In, Out, and Weight values are the same as regular neural networks. Enabled and Disabled genes are well, enabled and disabled. The innovation value is possibly the most defining feature of NEAT, since it allows for crossover of different topologies and historical tracking of each connection.
NEAT can mutate or change both its weights and connections, so for example, Parent1 and Parent2 has 5 of the same connections, represented by innovation / ID numbers 1 through 5. Since they have the same connection nodes, the genetic algorithm will randomly pick either Parent1 weight or Parent2 weight. The excess and disjoint genes are inherited from the more fit parent.
NEAT will then mutate each genome, shown in the image below.
2
$\begingroup$
$\endgroup$
I haven't read any relevant paper about this, but I have seen some implementations based on what you are describing, arbitrarily called DGNN (Dynamic Growing Neural Network).
Hope this term can help your search.
