Return to Answer

where we may have $m \gg n$ (although this is not a strict requirement), i.e. you may have a lot more unlabelled data than labelled data (this can easily be the case, given that, in general, manual data annotation is expensive/laborious). Let's say that your ultimate task is to perform object recognition (or classification). Let's call this task the downstream task. So, you may think that $x_i$ and $u_i$ are images and $y_i$ are labels, like "cat" or "dog".

• In step 1, you have the labels that are generated automatically. But how are these labels generated? As I said, there are many ways. Let me describe one way (among many others!). Let's say that your unlabelled dataset $U$ contains high-resolution images (i.e. $u_i \in U$ are high-resolution images), then you could define your pre-text task as follows. You lower the resolution of your high-resolution images to create other images. Let $v_i$ be the low-resolution image created from the high-resolution image $u_i \in U$, then the training pair to your neural network $M$ is $(v_i, u_i) \in U'$, where $u_i$ is the label (which is the original high-resolution image) and $U'$ the labeled dataset automatically generated (i.e. with the algorithm I've just explained).

  So, these labels $u_i$ (high-resolution images) are semantically different than $y_i$ ("cat" or "dog") in the pairs $(x_i, y_i) \in D$. They are different because, here, we want to learn representations and not to perform object recognition/classification: the idea is that, by solving this pre-text task, your final trained neural network, should have learned features of the images in the unlabelled data (i.e. representation learning). These learned features can then be used to bootstrap training in the downstream task.

• In step 2, you use the labeled dataset $D$, which has been (typically) annotated (or labeled) by a human. This dataset contains pairs $(x_i, y_i)$, where $y_i$ is, for example, the label "cat".

  In this step, the pre-trained model $M$, with the SSL technique, can be fine-tuned with $D$ in a supervised fashion. Given that we start with a pre-trained model $M$, we are effectively performing transfer learning.

where we may have $m \gg n$ (although this is not a strict requirement), i.e. you may have a lot more unlabelled data than labelled data (this can easily be the case, given that, in general, manual data annotation is expensive/laborious). Let's say that your ultimate task is to perform object recognition (or classification). Let's call this task the downstream task. So, you may think that $x_i$ and $u_i$ are images and $y_i$ are labels, like "cat" or "dog" (let's say that you want to differentiate between cats and dogs).

• In step 1, you have the labels that are generated automatically. But how are these labels generated? As I said, there are many ways. Let me describe one way (among many others!). Let's say that your unlabelled dataset $U$ contains high-resolution images (i.e. $u_i \in U$ are high-resolution images), then you could define your pre-text task as follows. You lower the resolution of your high-resolution images to create other images. Let $v_i$ be the low-resolution image created from the high-resolution image $u_i \in U$, then the training pair to your neural network $M$ is $(v_i, u_i) \in U'$, where $u_i$ is the label (which is the original high-resolution image) and $U'$ the labeled dataset automatically generated (i.e. with the algorithm I've just explained).

  So, these labels $u_i$ (high-resolution images) are semantically different than $y_i$ ("cat" or "dog") in the pairs $(x_i, y_i) \in D$. They are different because, here, we want to learn representations and not to perform object recognition/classification: the idea is that, by solving this pre-text task, your final trained neural network, should have learned features of the images in the unlabelled data (i.e. representation learning). These learned features can then be used to bootstrap training in the downstream task.

• In step 2, you use the labeled dataset $D$, which has been typically annotated (or labeled) by a human. As stated above, this dataset contains pairs $(x_i, y_i)$, where $y_i$ is, for example, the label "cat" or "dog".

  In this step, the pre-trained model $M$, with the SSL technique, can be fine-tuned with $D$ in a supervised fashion. Given that we start with a pre-trained model $M$, we are effectively performing transfer learning.

1. Self-supervised learning (SSL): learn representations of your images $u_i \in U$ by training a neural network $M$ with $U$ to solve a so-called pretext (or auxiliary task); there are many pre-text tasks: you can find many examples here, here and here (see example below too);

2. Supervised learning (SL) by transfer learning: fine-tune $M$ with $D$ (the labeled dataset), in a supervised way; this task solved with SL is the so-called downstream task

• In step 1, you have the labels that are generated automatically. But how are these labels generated? As I said, there are many ways. Let me describe one way (among many others!). Let's say that your unlabelled dataset $U$ contains high-resolution images (i.e. $u_i \in U$ are high-resolution images), then you could define your pre-text task as follows. You lower the resolution of your high-resolution images. Let $v_i$ be the low-resolution image created from the high-resolution image $u_i \in U$, then the training pair to your neural network $M$ is $(v_i, u_i) \in U'$, where $u_i$ is the label (which is the original high-resolution image) and $U'$ the labeled dataset automatically generated (i.e. with the algorithm I've just explained).

  So, these labels $u_i$ (high-resolution images) are semantically different than $y_i$ ("cat" or "dog") in the pairs $(x_i, y_i) \in D$. They are different because, here, we want to learn representations and not to perform object recognition/classification: the idea is that, by solving this pre-text task, your final trained neural network, should have learned features of the images in the unlabelled data (i.e. representation learning). These learned features can then be used to bootstrap training in the downstream task.

• In step 2, you use the labeled dataset $D$, which has been (typically) annotated (or labeled) by a human. This dataset contains pairs $(x_i, y_i)$, where $y_i$ is, for example, the label "cat".

  In this step, the pre-trained model $M$, with the SSL technique, can be fine-tuned with $D$ in a supervised fashion. Given that we start with a pre-trained model $M$, we are effectively performing transfer learning.

If this answer is still unclear, maybe you should have a look at existing implementations of SSL techniques for more inspiration.

1. Self-supervised learning (SSL): learn representations of your images $u_i \in U$ by training a neural network $M$ with $U$ to solve a so-called pretext (or auxiliary task); there are many pre-text tasks: you can find many examples here, here and here (see example below too);

2. Supervised learning (SL) by transfer learning: fine-tune $M$ with $D$ (the labeled dataset), in a supervised way; this task is known as downstream task (as stated above)

• In step 1, you have the labels that are generated automatically. But how are these labels generated? As I said, there are many ways. Let me describe one way (among many others!). Let's say that your unlabelled dataset $U$ contains high-resolution images (i.e. $u_i \in U$ are high-resolution images), then you could define your pre-text task as follows. You lower the resolution of your high-resolution images to create other images. Let $v_i$ be the low-resolution image created from the high-resolution image $u_i \in U$, then the training pair to your neural network $M$ is $(v_i, u_i) \in U'$, where $u_i$ is the label (which is the original high-resolution image) and $U'$ the labeled dataset automatically generated (i.e. with the algorithm I've just explained).

  So, these labels $u_i$ (high-resolution images) are semantically different than $y_i$ ("cat" or "dog") in the pairs $(x_i, y_i) \in D$. They are different because, here, we want to learn representations and not to perform object recognition/classification: the idea is that, by solving this pre-text task, your final trained neural network, should have learned features of the images in the unlabelled data (i.e. representation learning). These learned features can then be used to bootstrap training in the downstream task.

• In step 2, you use the labeled dataset $D$, which has been (typically) annotated (or labeled) by a human. This dataset contains pairs $(x_i, y_i)$, where $y_i$ is, for example, the label "cat".

  In this step, the pre-trained model $M$, with the SSL technique, can be fine-tuned with $D$ in a supervised fashion. Given that we start with a pre-trained model $M$, we are effectively performing transfer learning.

If this answer is still unclear, maybe you should have a look at existing implementations of SSL techniques (such as this) for more inspiration.

• In step 1, you have the labels that are generated automatically. But how are these labels generated? As I said, there are many ways. Let me describe one way (among many others!). Let's say that your unlabelled dataset $U$ contains high-resolution images (i.e. $u_i \in U$ are high-resolution images), then you could define your pre-text task as follows. You lower the resolution of your high-resolution images. Let $v_i$ be the low-resolution image created from the high-resolution image $u_i \in U$, then the training pair to your neural network $M$ is $(v_i, u_i) \in U'$, where $u_i$ is the label (which is the original high-resolution image) and $U'$ the labeled dataset automatically generated (i.e. with the algorithm I've just explained).

  So, these labels $u_i$ are semantically different than $y_i$ in the pairs $(x_i, y_i) \in D$. They are different because, here, we want to learn representations and not to perform object recognition/classification: the idea is that, by solving this pre-text task, your final trained neural network, should have learned features of the images in the unlabelled data (i.e. representation learning). These learned features can then be used to bootstrap training in the downstream task.

• In step 2, you use the labeled dataset $D$, which has been (typically) annotated (or labeled) by a human. This dataset contains pairs $(x_i, y_i)$, where $y_i$ is, for example, the label "cat".

  In this step, the pre-trained model $M$, with the SSL technique, can be fine-tuned with $D$ in a supervised fashion. Given that we start with a pre-trained model $M$, we are effectively performing transfer learning.

If this answer is still unclear, maybe you should have a look at existing implementations of SSL techniques for more inspiration.

(Note that SSL can also refer to something different than what has been explained in this answer. See my other answer for more details.)

• In step 1, you have the labels that are generated automatically. But how are these labels generated? As I said, there are many ways. Let me describe one way (among many others!). Let's say that your unlabelled dataset $U$ contains high-resolution images (i.e. $u_i \in U$ are high-resolution images), then you could define your pre-text task as follows. You lower the resolution of your high-resolution images. Let $v_i$ be the low-resolution image created from the high-resolution image $u_i \in U$, then the training pair to your neural network $M$ is $(v_i, u_i) \in U'$, where $u_i$ is the label (which is the original high-resolution image) and $U'$ the labeled dataset automatically generated (i.e. with the algorithm I've just explained).

  So, these labels $u_i$ (high-resolution images) are semantically different than $y_i$ ("cat" or "dog") in the pairs $(x_i, y_i) \in D$. They are different because, here, we want to learn representations and not to perform object recognition/classification: the idea is that, by solving this pre-text task, your final trained neural network, should have learned features of the images in the unlabelled data (i.e. representation learning). These learned features can then be used to bootstrap training in the downstream task.

• In step 2, you use the labeled dataset $D$, which has been (typically) annotated (or labeled) by a human. This dataset contains pairs $(x_i, y_i)$, where $y_i$ is, for example, the label "cat".

  In this step, the pre-trained model $M$, with the SSL technique, can be fine-tuned with $D$ in a supervised fashion. Given that we start with a pre-trained model $M$, we are effectively performing transfer learning.

Note that SSL can also refer to something (slightly) different than what has been explained in this answer. See my other answer for more details. Moreover, note that you can perform representation learning with SSL without necessarily solving a downstream task later, which may also not be an SL task (in the example above, I've described a downstream task that is an SL task only for simplicity).

If this answer is still unclear, maybe you should have a look at existing implementations of SSL techniques for more inspiration.