11

I'd like to add to what's been said already that your question touches upon an important notion in machine learning called transfer learning. In practice, very few people train an entire convolutional network from scratch (with random initialization), because it is time consuming and relatively rare to have a dataset of sufficient size. Modern ConvNets ...


6

Here is one way you could do that. After training your network, you can save its weights to disk. This allows you to load this weights when new data becomes available and continue training pretty much from where your last training left off. However, since this new data might come with additional classes, you now do pre-training or fine-tuning on the network ...


6

One application I know of being used in industry is of image classification, by only training the last layer of one of the inception models released by Google, with the desired number of classes. I can't provide specific details. Transfer learning is useful when: You do not have the resources (time, processing power, etc.) to train a DL model from ...


6

Learning without Forgetting (LwF) is an incremental learning (sometimes also called continual or lifelong learning) technique for neural networks, which is a machine learning technique that attempts to avoid catastrophic forgetting. There are several incremental learning approaches. LwF is an incremental learning approach based on the concept of ...


5

They are all related terms. From top to bottom: One-shot learning aims to achieve results with one or very few examples. Imagine an image classification task. You may show an apple and a knife to a human and no further examples are needed to continue classifying. That would be the ideal outcome, but for algorithms. In order to achieve one-shot learning (...


5

Meta-learning is more about speeding up and optimizing hyperparameters for networks that are not trained at all, whereas transfer learning uses a net that has already been trained for some task and reusing part or all of that network to train on a new task which is relatively similar. So, although they can both be used from task to task to a certain degree, ...


5

In terms of transfer learning, semantic gap means different meanings and purposes behind the same syntax between two or more domains. For example, suppose that we have a deep learning application to detect and label a sequence of actions/words $a_1, a_2, \ldots, a_n$ in a video/text as a "greeting" in a society A. However, this knowledge in Society ...


4

Why is ImageNet so popular for transfer learning? Models pre-trained on the ImageNet datasets have been the de-facto choice for many years now. Many popular reasons as to why people think that ImageNet is so effective for transfer learning are the following: ImageNet is a truly large-scale dataset that contains over 1 million images, each of which has a ...


4

Use Fine Tuning You can simply use a pre-trained model on ImageNet, as this data set has multiple snakes classes. Then you can fine tune the model with your own small data set and outputs. See this for further understanding : Fine Tuning in Keras (if you don't use Keras, there are other tutorials on the internet using other Machine Learning framework) ...


3

First of all, I would like to say that it is possible that these terms are used inconsistently, given that at least transfer learning, AFAIK, is a relatively new expression, so, the general trick is to take terminology, notation and definitions with a grain of salt. However, in this case, although it may sound confusing to you, all of the current ...


3

When doing transfer learning it makes sense to have different update policies for "inherited" parameters and the "new" parameters. "Inherited" parameters are pre-trained on dataset1 and they typically form the front end of the deep model. The "new" parameters are trained from scratch and they typically produce the desired predictions on dataset2. It would be ...


3

Besides using transfer learning described in other answer, you should consider using siamese network. This type of network is used in cases when one does not posess many examples of objects he wants to distinguish. General idea is that instead of "telling" the network "This is a cobra", you provide information like: "This is a cobra, and that is a ...


3

Yes, this is quite the expected behavior. The main difference between expected and current behavior lies in the amount of data you are using for training VS the amount of data that the pre-trained model was trained with. Take into account that pre-trained models have been trained over popular datasets, the most common ones are: COCO, ImageNet and Open Images....


2

Unsupervised pre-training was done only very shortly, as far as I know, at the time when deep learning started to actually work. It extracts certain regularities in the data, which a later supervised learning can latch onto, so it is not surprising that it might work. On the other hand, unsupervised learning doesn't give particularly impressive results in ...


2

Disclosure: I am a product manager on Google Cloud Platform. [...] why does everyone have to repeat the effort of learning the same things? If Google has already learned cats, or if someone already has a program to recognize handwritten digits, can this knowledge be shared and re-used? Or is it just a matter of paying for them? You don't have to ...


2

What I want to achieve is incremental training. So, as soon as I get new data, I can further train my already trained model and I don't have to retrain everything. Learning without forgetting is one of the methods to solve multitask learning. If your model trained to solve problem A and then after sometimes you need your model to solve new problem B without ...


2

ResNet is an architecture for object recognition and you may use it to do your classification task. Fast RCNN may improve your results but is a more difficult architecture to implement. If you want to go in this direction the best place to start is the arxiv paper of the Fast R-CNN (arxiv.org/abs/1504.08083). If I am not wrong, there is an implementation ...


2

The difference really comes down to the fact that in meta-learning, there is a population of tasks $\tau$ which have distribution $p(\tau)$. The goal is to perform well on a task drawn from $p(\tau)$. Generally 'perform well' means that with only a few training steps or data points, the model can give good classification accuracy, achieve high reward in an ...


2

Is it possible to use SSL to pre-train e.g. a faster R-CNN on a pretext task (for example, rotation), then use this pre-trained model for instance segmentation with the aim to get better accuracy? Yes, it's possible and this has already been done. I don't know the details (because I have not yet read those papers), but I will provide you with some links to ...


2

Neural networks use random number generators in multiple places. Most notably for weight initialisation, but also for features such as dropout, selecting minibatches within epochs, and train/cv split for cross-validation. That means that any result metric from the neural network e.g. accuracy, loss, F1 score, is a random variable. Reporting a single value of ...


1

It seems like transfer learning is only applicable to neural networks. Is this a correct assumption? No. Wiki page give you pointers of several examples in other methodologies. While I was looking for examples of Transfer Learning, most seemed to be based on image data, audio data, or text data. I was not able to find an example of training a neural ...


1

I think the answer is yes, this problem is known as "Where To transfer", please refer to this paper Learning What and Where to Transfer to know what I am talking about, and correct me if I understand something wrongly.


1

No, transfer learning cannot be applied "between" different architectures, as transfer learning is the practice of taking a neural network that has already been trained on one task and retraining it on another task with the same input modality, which means that only the weights (and other trainable parameters) of the network change during transfer ...


1

Why is this layer freezing required? It's not. What are the effects of layer freezing? The consequences are: (1) Should be faster to train (the gradient will have far less components) (2) Should require less data to train on If you do unfreeze the weights, I'd think your performance would be better because you are adjusting (i.e., fine-tuning) the parameters ...


1

I think there is no simple way to transfer knowledge changes between different models. If you take your initial model and create a new version of it which you use to learn some other task (like "Walk to a specific location"), then the values copied from the first (original) model change in the second model. From that moment on, training the former ...


1

The model has learnt the "features" for the type of inputs, eg. faces. For the problem to be called one-shot, it needs to also correctly classify/compare any new samples. For example, in face recognition application, any new person's images should also result a positive for their own image and negative for any other seen or unseen image. Since we are using ...


1

Even if you want to re-train your model for just one new class you will have to prepare your training data such that it includes all or most of the classes which you want to predict. Most of the times last two layers of a network have the data of number of labels which are to be predicted and that should always be sum of the number of classes you already ...


1

Transfer learning is a field where you apply knowledge from a source onto a target. This is a vague notion and there is an abundance of literature pertaining to it. Given your question I will work under the assumption that you are referring weight/architecture sharing between model (in other words training a model on one dataset and using it as a featurizer ...


1

For the vast majority of cases where you have a dynamic(and assumed non-linear) relationship between your input and output, you would not use modified architecture. You would simply retrain on the new data. In some cases, based on domain knowledge or intuition, one might put a "weight" on the new data to increase or decrease its importance relative to ...


1

Adam reduces the learning rate over time. When you change to the new training data, you want to reset the learning rate. But Adam might not be the best choice for the second round of training - it can make big changes to the inherited weights, which prevents the transfer of previous learning. It can be good to switch to simple SGD for the second round.


Only top voted, non community-wiki answers of a minimum length are eligible