Iteratively and adaptively increasing the network size during training

Question

For an experiment that I'm working on, I want to train a deep network in a special way. I want to initialize and train a small network first, then, in a specific way, I want to increase network depth leading to a bigger network which is subsequently to be trained. This process will be repeated until one reaches the desired depth.

It would be great if anybody heard of anything similar and could point out to me some related work. I think in some paper I read something about a related technique where people used something similar, but I don't find it anymore.

Answer 1

Neuroevolution Through Augmenting Topologies or NEAT may be what you are referring to. The original paper by Kenneth O. Stanley is here

NEAT combines a neural network and a genetic algorithm. Instead of using back propagation or gradient descent to "train" your network, NEAT creates a population of very simple neural networks (no connections) and evolves them with fitness evaluation, crossover, and mutation. The genome syntax: every connection gene has a few settings. In node, Out node, Weight of connection, activated, and innovation. In, Out, and Weight values are the same as regular neural networks. Enabled and Disabled genes are well, enabled and disabled. The innovation value is possibly the most defining feature of NEAT, since it allows for crossover of different topologies and historical tracking of each connection.NEAT can mutate or change both its weights and connections, so for example, Parent1 and Parent2 has 5 of the same connections, represented by innovation / ID numbers 1 through 5. Since they have the same connection nodes, the genetic algorithm will randomly pick either Parent1 weight or Parent2 weight. The excess and disjoint genes are inherited from the more fit parent. NEAT will then mutate each genome, shown in the image below.

Answer 2

I haven't read any relevant paper about this, but I have seen some implementations based on what you are describing, arbitrarily called DGNN (Dynamic Growing Neural Network).

Hope this term can help your search.

Answer 3

At DeepChess they trained stacked autoencoders:

Training Pos2Vec: We first trained a deep belief network (DBN), which would later serve as the initial weights for supervised training. The DBN is based on stacked autoencoders which are trained using layer-wise unsupervised training. The network consists of five fully connected layers of sizes: 773–600–400–200–100. We initially trained the first layer (i.e., a 3-layer (773–600–773) autoencoder), before fixing its weights and training the weights of a new (600–400–600) autoencoder, and so on.

I suppose this matches your criteria, although the network's architecture isn't very flexible. They don't introduce skip-connections etc.