Would this work at all?
Idea is to start training a neural net with some number of nodes. Then, add some new nodes and more layers and start training only the new nodes (or only modifying the old nodes very slightly). Ideally, we would connect all old nodes to the new layer added since we might have learned many useful things in the hidden layers. Then repeat this many times.
Intuition is that if the old nodes give bad information the new layer of nodes will weight the activations of old nodes close to zero and learn new/better concepts in the new nodes. The benefit is that we will keep old knowledge forever.
Caveat is that the network can still temporarily "forget" concepts if a new layer weights old information close to zero, but it can potentially remember it again too.
If this completely fails, I'm curious if there's some known way to prevent a neural network from forgetting concepts it learned.