I have two datasets, Dataset 1(D1) and Dataset 2(D2). D1 has around 22000 samples, and D2 has around 8000 samples. What I am doing is that I train a Deep Neural Network model with around three layers on the dataset D1, which has an accuracy of around 84% (test accuracy = 82%).
Now, when I use that model to make predictions on D2 without any fine-tuning or anything, I get an accuracy of around 15%(test accuracy = 12.3%). But when I add three more layers to the pre-trained model while keeping the three layers of the initial model(trained on D1) frozen, I get around 90% accuracy (test accuracy = 87.6%) on D2.
This tells me that because the initial model was performing so poorly without any fine-tuning, most of the learning that led to the 90% accuracy was only because of the additional layers, not the layers that were transferred from the model trained on the D1 dataset. Is this a correct inference? And if it is, then is it still valid to call this a Transfer Learning application? Or does it has to have more accuracy without fine-tuning to be rightly listed as a Transfer Learning problem.