2
$\begingroup$

I have two datasets, Dataset 1(D1) and Dataset 2(D2). D1 has around 22000 samples, and D2 has around 8000 samples. What I am doing is that I train a Deep Neural Network model with around three layers on the dataset D1, which has an accuracy of around 84% (test accuracy = 82%).

Now, when I use that model to make predictions on D2 without any fine-tuning or anything, I get an accuracy of around 15%(test accuracy = 12.3%). But when I add three more layers to the pre-trained model while keeping the three layers of the initial model(trained on D1) frozen, I get around 90% accuracy (test accuracy = 87.6%) on D2.

This tells me that because the initial model was performing so poorly without any fine-tuning, most of the learning that led to the 90% accuracy was only because of the additional layers, not the layers that were transferred from the model trained on the D1 dataset. Is this a correct inference? And if it is, then is it still valid to call this a Transfer Learning application? Or does it has to have more accuracy without fine-tuning to be rightly listed as a Transfer Learning problem.

$\endgroup$
4
  • $\begingroup$ Are the quoted accuracy values from training, cross validation or fully held out test data? Obviously the 15% was using D2 as a test set for the first model trained on D1. But what about the others? $\endgroup$ Jun 14 at 11:32
  • $\begingroup$ The reported accuracies are the training accuracies only. The validation accuracies are only 3-4% lower than the training accuracies. During training on the D1 dataset, I split it into 80-20 train test data and in the same ratio for D2 while fine-tuning on that dataset. $\endgroup$
    – Ravish Jha
    Jun 14 at 13:20
  • $\begingroup$ Probably worth quoting the test accuracies and noting they are such with aquick edit to the question. If the only thing we had to go on was training accuracies then your experiment looks a lot like overfitting to D1 then to D2, and it is more difficult to offer clear advice because an asnwer would need to address that point too $\endgroup$ Jun 14 at 13:31
  • $\begingroup$ Did as suggested. The network wasn't overfitting so I didn't include the test accuracies earlier. $\endgroup$
    – Ravish Jha
    Jun 14 at 16:48
1
$\begingroup$

This tells me that because the initial model was performing so poorly without any fine-tuning, most of the learning that led to the 90% accuracy was only because of the additional layers, not the layers that were transferred from the model trained on the D1 dataset. Is this a correct inference?

This is a possibility, but not the only one. If you were re-purposing a classifier for ImageNet classes to specific new image type, or even the same classes but with the labels in a different order, then this large initial drop in accuracy would be expected.

The transfer learning could be helping in two different measurable ways:

  • The training for the new purpose is faster (fewer epochs required) than if done from scratch with a brand new network.

  • The end accuracy is better than could be achieved with just D2 dataset and a brand new network.

The only way to tell if either of these are the case is to compare results by using just the normal D2 features and a re-initialised copy of the original NN used to learn from D1 (by that I mean adapt your initial training script from D1 to work with D2 - changing the dataset file names and the ouput layer shape should be all you need to do). Look at the learning curve for this training - if it is significantly slower or worse accuracy at the end, then the transfer learning has made a difference.

And if it is, then is it still valid to call this a Transfer Learning application? Or does it has to have more accuracy without fine-tuning to be rightly listed as a Transfer Learning problem.

I am not sure it matters if the results using transfer learning are worse. It is - at least in my opinion - an attempt to use transfer learning.

If transfer learning has not produced better results, then you will maybe have demonstrated that there is not enough overlap from the D1 dataset and problem to the D2 ones to justify using transfer learning in your use case.

This result may also depend on the sizes of datasets D1 and D2, even if your experiment stays in the same problem domain. The number of examples in D1 is ~3 times the size of D2, which is not much of a difference compared to transfer learning done using large pre-trained image or language models.

$\endgroup$
2
  • $\begingroup$ That is a really nice explanation, and it made my mind clear about it a lot. Can you explain, "The only way to tell if either of these are the case is to compare results by using just the normal D2 features and a re-initialised copy of the original NN used to learn from D1." a bit more, please? $\endgroup$
    – Ravish Jha
    Jun 14 at 16:33
  • $\begingroup$ @RavishJha. Sure. Your inputs for D1 and D2 are the same shape, so you can use the same neural network architecture for D2 as D1, except maybe the last layer if there are different number of classes. It makes sense to do this for a fair comparison - if you changed the number of hidden layers or other hyperparameters, it might be that change which made a difference in accuracy. The plan is train for D2 in the same way as you trained for D1 - without any transfer learning. You should be able to use almost the same training scripts etc, so there is not much development time needed. $\endgroup$ Jun 14 at 18:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.