This article on Dynamically Expandable Neural Networks (DEN) (by Harshvardhan Gupta) is based on this paper Lifelong Learning with Dynamically Expandable Networks (by Jeongtae Lee, Jaehong Yoon, Eunho Yang, Sung Ju Hwang)
This presents 3 solutions to increase the capacity of the network if needed retaining whatever useful information from the old model and train the new model:
- Selective retraining
- Dynamic Network Expansion
- Network Split/Duplication
To me, it seems that such neural network is dynamic and improving. As such, they answer partially your question. If they don't sorry about that.
