Is it possible to use self-supervised learning on different images for the pretext and downstream tasks?

Question

I have just come across the idea of self-supervised learning. It seems that it is possible to get higher accuracies on downstream tasks when the network is trained on pretext tasks.

Suppose that I want to do image classification on my own set of images. I have limited data on these images and maybe I can use self-supervised learning to achieve better accuracies on these limited data.

Let's say that I try to train a neural network on a pretext task of predicting the patch position relative to the center patch on different images that are readily available in quantity, such as cats, dogs, etc.

If I try to initialise the weights of my neural network, then do image classification on my own images, which are vastly different from that of the images used in the pretext task, would self-supervised learning work because the images for the pretext and downstream tasks are different?

TLDR: Must the images used in the pretext task and the downstream tasks be the same?

Answer 1

No, the images do not need to be same. You can use different images for downstream task however you need to do some changes in model definition while loading model state_dict as CNN architecture used for pretext task 'relative patch location', assume Alexnet will expect 2 patches as input.