This article on [Dynamically Expandable Neural Networks][1]Dynamically Expandable Neural Networks (DEN) (by Harshvardhan Gupta) is based on this paper: [Lifelong Learning with Dynamically Expandable Networks][2] Lifelong Learning with Dynamically Expandable Networks (by Jeongtae Lee, Jaehong Yoon, Eunho Yang, Sung Ju Hwang)
This presents 3 solutions to increase the capacity of the network if needed retaining whatever useful information from the old model and train the new model:
- Selective retraining
- Dynamic Network Expansion
- Network Split/Duplication
To me, it seems that such neural network is dynamic and improving. As such, they answer partially your question. If they don't sorry about that.
[1]: https://hackernoon.com/dynamically-expandable-neural-networks-ce75ff2b69cf by Harshvardhan Gupta 2017-09-18
[2]: https://arxiv.org/pdf/1708.01547v2.pdf by Jeongtae Lee, Jaehong Yoon, Eunho Yang, Sung ju hwang 2017-09-11
