3
$\begingroup$

There are many types of CNN architectures: LeNet, AlexNet, VGG, GoogLeNet, ResNet, etc. Can we apply transfer learning between any two different CNN architectures? For instance, can we apply transfer learning from AlexNet to GoogLeNet, etc.? Or even just from a "conventional" CNN to one of these other architectures, or the other way around? Is this possible in general?

EDIT: My understanding is that all machine learning models have the ability to perform transfer learning. If this is true, then I guess the question is, as I said, whether we can transfer between two different CNN architectures – for instance, what was learned by a conventional CNN to a different CNN architecture.

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Do you know what "transfer learning" usually refers to in machine learning? $\endgroup$
    – nbro
    Sep 20, 2020 at 17:46
  • $\begingroup$ @nbro I’m studying it now. Apparently there are four categories of “transfer learning”, each involving a transfer of some different aspect of the neural network. Is that correct? $\endgroup$
    – The Pointer
    Sep 20, 2020 at 17:48
  • $\begingroup$ I only know one form of learning that people usually refer to as "transfer learning", but maybe there are others. Can you please link us to the article or resource you are reading that tells you that there are 4 different forms of transfer learning? $\endgroup$
    – nbro
    Sep 20, 2020 at 17:49
  • $\begingroup$ @nbro This isn’t the resource I am studying, but you can find the categories in this medium.com/georgian-impact-blog/… article. $\endgroup$
    – The Pointer
    Sep 20, 2020 at 17:53
  • $\begingroup$ Ok, thanks for sharing it with us. $\endgroup$
    – nbro
    Sep 20, 2020 at 17:55

2 Answers 2

Reset to default
1
$\begingroup$

No, transfer learning cannot be applied "between" different architectures, as transfer learning is the practice of taking a neural network that has already been trained on one task and retraining it on another task with the same input modality, which means that only the weights (and other trainable parameters) of the network change during transfer learning but not the architecture.

In my understanding, transfer learning is also only really effective in deep learning, but I could be wrong, considering that this Google search seems to yield some results.

You might otherwise be thinking of knowledge distillation, which is a related but different concept, where an already trained network acts as a teacher and teaches another network (a student network) with possibly a different architecture (or a machine learning model not based on neural networks at all) the correct outputs for a bunch of input examples.

Improve this answer
$\endgroup$
1
  • $\begingroup$ "Transfer learning and machine learning are closely related. On one hand, the aim of transfer learning encompasses that of machine learning in that its key in- gredient is “generalization.” In other words, it explores how to develop general and robust machine learning models that can apply to not only the training data, but also unanticipated future data. Therefore, all machine learning models should have the ability to conduct transfer learning." Chapter 1.3 Relationship to Existing Machine Learning Paradigms of Transfer Learning by Yang, Zhang, Dai, and Pan. $\endgroup$
    – The Pointer
    Sep 22, 2020 at 3:16
1
$\begingroup$

I think the answer is yes, this problem is known as "Where To transfer", please refer to this paper Learning What and Where to Transfer to know what I am talking about, and correct me if I understand something wrongly.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .