17
$\begingroup$

I am fairly new to Neural Networks and have a requirement where the output classes are not defined from the start.

More and more classes will be introduced later on based on incoming data.

This means that every time I introduce a new class I would need to retrain...

I have just worked on ANN so don't know much depth of other techniques.

Is there a way where I can deferentially train the model without touching the old model much..??

$\endgroup$
  • $\begingroup$ Interesting problem. Welcome to AI! $\endgroup$ – DukeZhou Sep 6 '17 at 18:30
8
$\begingroup$

I'd like to add to what's been said already that your question touches upon an important notion in machine learning called transfer learning. In practice, very few people train an entire convolutional network from scratch (with random initialization), because it is time consuming and relatively rare to have a dataset of sufficient size.

Modern ConvNets take 2-3 weeks to train across multiple GPUs on ImageNet. So it is common to see people release their final ConvNet checkpoints for the benefit of others who can use the networks for fine-tuning. For example, the Caffe library has a Model Zoo where people share their network weights.

When you need a ConvNet for image recognition, no matter what your application domain is, you should consider taking an existing network, for example VGGNet is a common choice.

There are a few things to keep in mind when performing transfer learning:

  • Constraints from pretrained models. Note that if you wish to use a pretrained network, you may be slightly constrained in terms of the architecture you can use for your new dataset. For example, you can’t arbitrarily take out Conv layers from the pretrained network. However, some changes are straight-forward: due to parameter sharing, you can easily run a pretrained network on images of different spatial size. This is clearly evident in the case of Conv/Pool layers because their forward function is independent of the input volume spatial size (as long as the strides “fit”).

  • Learning rates. It’s common to use a smaller learning rate for ConvNet weights that are being fine-tuned, in comparison to the (randomly-initialized) weights for the new linear classifier that computes the class scores of your new dataset. This is because we expect that the ConvNet weights are relatively good, so we don’t wish to distort them too quickly and too much (especially while the new Linear Classifier above them is being trained from random initialization).

Additional reference if you are interested in this topic: How transferable are features in deep neural networks?

$\endgroup$
5
$\begingroup$

Fist of all, good question this! I also wonder how other people are actually doing this :).

Anyway here is one way you could do that:

After training your network, you can save its weights to disk. This allows you to load this weights when new data becomes available and continue training pretty much from where your last training left off. However since this new data might come with additional classes you now do Pre-training or Fine-tuning on the network with weights previously saved. The only thing you have to do at this point is make the last layer(s) accommodate the new classes that have now been introduced with the arrival of your new dataset, most importantly include the extra classes(eg if your last layer initially had 10 classes and now you have found 2 more classes, as part of your Pre-training/Fine-tuning, you replace it with 12 classes). In short, repeat this circle :

Repeat

$\endgroup$
  • $\begingroup$ if you only accommodate the new classes in the last layer ( training classes + new classes) the model cannot be fit because we want to train with the new classes (only) and the model expect an array with the shape of (training + the new classes, ). $\endgroup$ – Joel Carneiro Mar 20 at 11:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.