This article on [Dynamically Expandable Neural Networks][1] (DEN) (by Harshvardhan Gupta) is based on this paper [Lifelong Learning with Dynamically Expandable Networks][2] (by Jeongtae Lee, Jaehong Yoon, Eunho Yang, Sung Ju Hwang)

This presents 3 solutions to increase the capacity of the network if needed retaining whatever useful information from the old model and train the new model:

 - Selective retraining
 - Dynamic Network Expansion 
 - Network Split/Duplication

To me, it seems that such neural network is dynamic and improving. As such, they answer partially your question. If they don't sorry about that.

  [1]: https://hackernoon.com/dynamically-expandable-neural-networks-ce75ff2b69cf

  [2]: https://arxiv.org/pdf/1708.01547v2.pdf