What's the differences between semi-supervised learning and self-supervised visual representation learning, and how they are connected?
Semi-supervised learning is a combination of supervised and unsupervised learning. In semi-supervised learning, there are two datasets: a labelled one and an unlabelled one. There are two main problems that can be solved using semi-supervised learning: transductive learning and inductive learning (generalisation). In the case of transductive learning, the goal is to label the given unlabelled data. In the case of inductive learning, the goal is to find a function that maps inputs to outputs (like classification).
Self-supervised learning (SSL) is a special case of supervised learning where the training dataset is not manually labelled by a human, but the labels (for each training sample) are generated by exploiting correlations of the inputs or between different inputs (coming from different sensor modalities). For example, in the context of robotics, a robot can be equipped with different sensors. The output of these sensors can be correlated. For example, suppose a robot is equipped with a camera and a proximity sensor (that senses the presence of obstacles around the robot). The output of these sensors is clearly correlated (e.g. a proximity sensor detects an object when there is an object in the camera frame). There are thus several ways of performing self-supervised learning. It depends on the problem, the available sensors and data.
Self-supervised learning can also refer to an unsupervised learning that is used to find features of the unlabelled data (this task is called "representation learning"), which are used to "label" the same unlabelled data.
RL could actually be considered an instance of self-supervised learning, because the inputs (the experience) is used to label the quality of the states (e.g. by estimating the value function) or actions.
SSL has thus slightly different definitions depending on the context.
How are semi and self-supervised learning related? For example, self-supervised learning can be used to construct the labelled dataset that is used in semi-supervised learning.
The previous answer has given a good insight into the difference between two areas. I would like to give more examples.
Semi-Supervised Learning work with improving the data set by adding up new examples. There are iterative systems where we train a model on a given dataset and improve the model further after deploying it on the real world by adding interactions of the real world and their outcomes to further train the system.
Self-Supervised Learning is becoming a very hot topic these days. It has the ability to understand the underline properties of a given dataset with some kind of a supervisory signal (Not exactly a label). self-Attention introduced in Transformers is a modern day popular self-supervised learning. Also, check this tweet from Yann Lecun tweet
Both semi-supervised and self-supervised methods are similar in the sense that the goal is to learn with fewer labels per class. The way both formulate this is quite different:
- Self-Supervised Learning:
This line of work aims to learn image representations without requiring human-annotated labels and then use those learned representations on some downstream tasks. For example, you could take millions of unlabeled images, randomly rotate them by either 0, 90, 180 or 270 degrees and then train a model to predict the rotation angle. Once the model is trained, you can use transfer learning to fine-tune this model on a downstream task like cat/dog classification just like how you finetune ImageNet pretrained models. You can view an overview of the methods and also look at contrastive learning methods that are currently giving state-of-the-art results such as SimCLR and PIRL.
- Semi-supervised Learning
Different from self-supervised learning, semi-supervised learning aims to use both labeled and unlabeled data at the same time to improve the performance of a supervised model. An example of this is FixMatch paper where you train your model on labeled images. Then, for your unlabeled images, you apply augmentations to create two images for each unlabeled image. Now, we want to ensure that the model predicts the same label for both the augmentations of the unlabeled images. This can be incorporated into the loss as a cross-entropy loss.