Is continuous learning possible with a deep convolutional neural network, without changing its topology?

Question

In general, is continuous learning possible with a deep convolutional neural network, without changing its topology?

In my case, I want to use a convolutional neural network as a classifier of heartbeat types. The ECG signal is split, and a color image is created using feature extraction. These photos (the inputs) are fed into a deep CNN, but they must be labeled by someone first.

Are there ways to implement continuous learning in a deep neural network for image recognition? Does such an implementation make sense if the labels have to be specially prepared in advance?

Answer 1

In general, is continuous learning possible with a deep convolutional neural network, without changing its topology?

Your intuition that it is possible to perform incremental (aka continual, continuous or lifelong) learning by changing the NN's topology is correct. However, dynamically adapting the NN's topology is just one approach to continual learning (a specific example of this approach is DEN). So, there are other approaches, such as

• regularization approaches (examples are LwF, EWC and SI)

• ensemble approaches (examples are Learn++, FEL and PathNet)

• rehearsal approaches (an example is iCaRL)

For more details about these and other approaches (and problems related to continual learning and catastrophic forgetting in neural networks), take a look at this very nice review of continual learning approaches in neural networks. You should also check this answer.

Are there ways to implement continuous learning in a deep neural network for image recognition?

Yes. Many of the approaches focus on image recognition and classification, and often the experiments are performed on MNIST or similar datasets (e.g. see this paper).

Does such an implementation make sense if the labels have to be specially prepared in advance?

Yes, you can prepare your dataset in advance, and then later train incrementally (in fact, in the experiments I have seen in some of these papers, they usually do this to simulate the continual learning scenario), but I am not sure about the optimality of this approach. Maybe with batch learning (i.e. the usual offline learning where you train on all data), you would achieve higher performance.