What's the differences between semi-supervised learning and self-supervised visual representation learning, and how they are connected?
The previous answer has given a good insight into the difference between two areas. I would like to give more examples.
Semi-Supervised Learning work with improving the data set by adding up new examples. There are iterative systems where we train a model on a given dataset and improve the model further after deploying it on the real world by adding interactions of the real world and their outcomes to further train the system.
Self-Supervised Learning is becoming a very hot topic these days. It has the ability to understand the underline properties of a given dataset with some kind of a supervisory signal (Not exactly a label). self-Attention introduced in Transformers is a modern day popular self-supervised learning. Also, check this tweet from Yann Lecun tweet
Both semi-supervised and self-supervised methods are similar in the sense that the goal is to learn with fewer labels per class. The way both formulate this is quite different:
- Self-Supervised Learning:
This line of work aims to learn image representations without requiring human-annotated labels and then use those learned representations on some downstream tasks. For example, you could take millions of unlabeled images, randomly rotate them by either 0, 90, 180 or 270 degrees and then train a model to predict the rotation angle. Once the model is trained, you can use transfer learning to fine-tune this model on a downstream task like cat/dog classification just like how you finetune ImageNet pretrained models. You can view an overview of the methods and also look at contrastive learning methods that are currently giving state-of-the-art results such as SimCLR and PIRL.
- Semi-supervised Learning
Different from self-supervised learning, semi-supervised learning aims to use both labeled and unlabeled data at the same time to improve the performance of a supervised model. An example of this is FixMatch paper where you train your model on labeled images. Then, for your unlabeled images, you apply augmentations to create two images for each unlabeled image. Now, we want to ensure that the model predicts the same label for both the augmentations of the unlabeled images. This can be incorporated into the loss as a cross-entropy loss.
Semi-supervised learning is a combination of supervised and unsupervised learning. In semi-supervised learning, there are two datasets: a labelled one and an unlabelled one. There are two main problems that can be solved using semi-supervised learning: transductive learning and inductive learning (generalisation). In the case of transductive learning, the goal is to label the given unlabelled data. In the case of inductive learning, the goal is to find a function that maps inputs to outputs (like classification).
Self-supervised learning (SSL) is a supervised learning technique where the supervisory signal is automatically generated. In this answer, I provide a more complete description of SSL in different contexts, such as robotics, representation learning, and neural networks, so I suggest that you read it!
How are semi and self-supervised learning related? For example, self-supervised learning can be used to construct the labelled dataset that is used in semi-supervised learning.