What is the difference between self-supervised and unsupervised learning? The terms logically overlap (and maybe self-supervised learning is a subset of unsupervised learning?), but I cannot pinpoint exactly what that difference is.
What are the commonly agreed-upon 'definitions' of these terms? What is an example of unsupervised learning that is definitely not self-supervised learning?
In self-supervised learning (SSL) you use your own inputs $x$ (or a modification, e.g. a crop or with data augmentation applied) as the supervision. Instead, in unsupervised learning (UL) there is no supervision at all.
To clarify, both SSL and UL have in common the fact that the targets are missing. UL has no explicit supervision, while SSL replaces the targets with the inputs $x$, recovering 'supervision'.
An important thing is not to confuse self-supervised with semi-supervised or weakly-supervised: the latter two (semi- and weak-) refer to the fact that in a dataset $D$ some examples $x$ are not labeled, but the $y$ exist.
So, you can see SSL at the intersection between supervised and unsupervised learning. Actually, things got even more shaded in modern unsupervised deep learning methods that tend to mix approaches from both SSL and UL, like an AE that also have a density estimation head for example. Or even embeddings that are first learned by SSL and then fine-tuned for clustering in an unsupervised manner.
An unusual example is maybe unsupervised reinforcement learning, in which you maximize usually an entropy objective (e.g. on visited states) as a pre-training step to favor exploration.
What is an example of unsupervised learning that is definitely not self-supervised learning?
Density estimation, dimensionality reduction (e.g. PCA, t-SNE), and clustering (K-means), at least seen from a classical ML prospective are completely unsupervised: e.g. PCA tries just to preserve variance. Indeed, in DL things tend to blurry: e.g. you can use a V/AE for dimensionality reduction too.
While both methods learn from data without human-annotated labels, the primary difference lies in the way they use the data:
Input: "The quick brown fox jumps over the _____"
Prediction: "lazy"
Actual: "lazy"
In this case, the model is learning to predict the next word ("lazy") based on the input ("The quick brown fox jumps over the"). It's a self-supervised task because the label for training (the word "lazy") is part of the data itself.
For unsupervised learning, let's consider an example with text clustering. Suppose we have multiple sentences, including multiple instances of our sentence, and we want to group similar sentences together:
1. "The quick brown fox jumps over the lazy"
2. "A fast brown fox jumps over the lazy"
3. "The quick brown dog runs under the active"
4. "An energetic brown dog runs under the quick"
5. "The quick brown fox jumps over the lazy"
6. "A swift brown fox leaps over the lazy"
Clusters after unsupervised learning:
Cluster 1: 1, 2, 5, 6
Cluster 2: 3, 4
In this case, the model is learning to group similar sentences together without any explicit labels. Note that this is a highly simplified example, and real-world text clustering tasks would involve much more complex datasets and models.
Self-supervised learning is one approach to unsupervised learning. There are other approaches to unsupervised learning, too. In both cases, we have a dataset of instances with no labels, and we're trying to use them to learn a classifier.
Unsupervised learning includes any method for learning from unlabelled samples. Self-supervised learning is one specific class of methods to learn from unlabelled samples. Typically, self-supervised learning identifies some secondary task where labels can be automatically obtained, and then trains the network to do well on the secondary task.
One example of a secondary task is to predict whether an image has been flipped to be upside-down or not. Effectively, the assumption is that all (or most) natural photographs are taken by a photographer from an upright position, so we can automatically construct labels for all images in our dataset, as they're likely all right-side-up. (Moreover, we can create upside-down versions of all images in the dataset and automatically create labels of them: they're likely all upside-down.)
The concept behind self-supervised learning is that, hopefully, the classifier will have to learn something non-trivial about image semantics to be able to solve the secondary task. Thus, such a classifier might be a good starting point for the primary task, too.
Often, self-supervised learning is combined with supervised learning. For instance, we might have a small set of labelled images (labelled for the primary task we ultimately care about) and a large set of unlabelled images, and the classifier is trained to minimize a hybrid loss, which is the sum of a supervised loss on the labelled images and a self-supervised loss on the unlabelled images. This can be considered an instance of semi-supervised learning. Or, another example setting is that sometimes people learn a classifier on a large unlabelled dataset using self-supervised learning, then they fine-tune that classifier on a small labelled dataset for the primary task. This can be considered a form of transfer learning.
Empirically, this approach often leads to classifiers with better performance on the primary task than just using supervised learning on the labelled dataset. Or, it leads to classifiers that are more robust (to data corruption, distribution shift, etc.) than simply using supervised learning on the labelled dataset.
There are many forms of self-supervised learning, and techniques depend on the specific problem domain.
It's misleading to think of these as separate learning paradigms. Self-supervised learning IS an unsupervised learning algorithm that uses certain methods to derive learning signals without explicit labels.
