What is self-supervised learning in machine learning? How is it different from supervised learning?

  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$ – nbro Mar 6 at 1:12


The term self-supervised learning (SSL) has been used (sometimes differently) in different contexts and fields, such as robotics [1], reinforcement learning, deep learning [2], computer vision, natural language processing and neural networks. However, in all cases, the basic idea is to automatically generate some kind of supervisory signal to solve some task.

Below, I will provide a more specific definition, and an example that hopefully gives the intuition behind that definition, in the context of robotics, but the idea of automatically generating a supervisory signal is applicable to the other scenarios. How you generate that supervisory signal depends on the context. I will also briefly describe an example of SSL in the context of deep learning and neural networks.

Although the specific definition I will give is consistent with a paper that has been published by an important publisher, in general, terminology can be used inconsistently across different sources or fields, so it is possible that you will find another source or even research paper that uses the term self-supervised learning differently or maybe that uses another term to refer to SSL. If in doubt, you should take into account your context and use the definition given in your context.

Moreover, it is possible that other terms are used to refer to the same or very similar concepts. For example, the concept of distant supervision (DS) is very similar to SSL. It's not clear yet what all the differences and similarities between DS and SSL are (or if they are synonymous).

Self-supervised learning (in robotics)

In robotics, self-supervised learning is a (weakly) supervised learning technique where the training data is automatically (but often approximately) labeled. This can be done by finding and exploiting the relations or correlations between inputs coming from different sensor modalities.


To understand this definition, consider a robot that is equipped with a proximity sensor (which is a short-range sensor capable of detecting objects in front of the robot at short distances) and a camera (which is long-range sensor, but which does not provide a direct way of detecting objects). You can also assume that this robot is capable of performing odometry. An example of such a robot is Mighty Thymio.

Consider now the task of detecting objects in front of the robot at longer ranges than the range the proximity sensor allows. In general, we could train a CNN to achieve that. However, to train such CNN, in supervised learning, we would first need a labelled dataset, which contains labelled images (or videos), where the labels could e.g. be "object in the image" or "no object in the image". In supervised learning, this dataset would need to be manually labelled by a human, which clearly would require a lot of work.

To overcome this issue, we can use a self-supervised learning approach. In this example, the basic idea is to associate the output of the proximity sensors at a time step $t' > t$ with the output of the camera at time step $t$ (a smaller time step than $t'$).

More specifically, suppose that the robot is initially at coordinates $(x, y)$ (on the plane), at time step $t$. At this point, we still do not have enough info to label the output of the camera (at the same time step $t$). Suppose now that, at time $t'$, the robot is at position $(x', y')$. At time step $t'$, the output of the proximity sensor will e.g. be "object in front of the robot" or "no object in front of the robot". Without loss of generality, suppose that the output of the proximity sensor at $t' > t$ is "no object in front of the robot", then the label associated with the output of the camera (an image frame) at time $t$ will be "no object in front of the robot".

For more details about this specific example, have a look at the paper Learning Long-range Perception using Self-Supervision from Short-Range Sensors and Odometry (2019) by Mirko Nava et al., published in IEEE Robotics and Automation Letters.


Apart from the obvious advantage that there's no need for a human to manually label the dataset, another advantage and consequence of SSL is that it can more easily (with respect to plain supervised learning) be performed in an online fashion (given that data can be gathered and labelled without human intervention), where models can be updated or trained completely from scratch. Therefore, self-supervised learning should also be well suited for changing environments, data and, in general, situations.


SSL is often tailored to a specific robot and its sensors. Moreover, the generation of the labels may not be completely accurate, which could lead to other inaccuracies in the training of the model.

Other contexts

Deep learning

In deep learning, the term SSL has been used differently than in robotics. For example, in [4], two patches are randomly selected and cropped from an unlabelled image and the goal is to predict the relative position of the two patches. Of course, we have the relative position of the two patches once you have chosen them (i.e. we can keep track of their centres), so, in this case, this is the automatically generated supervisory signal. This example is similar to the example given in this other answer. This kind of task is sometimes known as a pretext or auxiliary task in the literature [11, 12, 13, 14].

Neural networks

Some neural networks, for example, autoencoders (AE) [10] are sometimes called self-supervised learning tools. In fact, you can train AEs without images that have been manually labelled by a human. More concretely, consider a de-noising AE, whose goal is to reconstruct the original image when given a noisy version of it. During training, you actually have the original image, given that you have a dataset of uncorrupted images and you just corrupt these images with some noise, so you can calculate some kind of distance between the original image and the noisy one, where the original image is the supervisory signal. In this sense, AEs are self-supervised learning tools, but it's more common to say that AEs are unsupervised learning tools, so SSL has also been used to refer to unsupervised learning techniques.

Further reading

There is a curated list of links to papers (and other resources) about SSL: https://github.com/jason718/awesome-self-supervised-learning. There's also this article, which attempts to compare SSL to other machine learning techniques. For other specific examples and definitions of SSL, you can read Digging Into Self-Supervised Monocular Depth Estimation or Multi-task Self-Supervised Visual Learning.

| improve this answer | |

Self-supervised learning is when you use some parts of the samples as labels for a task that requires a good degree of comprehension to be solved. I'll emphasize these two key points, before giving an example:

  • Labels are extracted from the sample, so they can be generated automatically, with some very simple algorithm (maybe just random selection).

  • The task requires understanding. This means that, in order to predict the output, the model has to extract some good patterns from the data, generating on the process a good representation.

A very common case for semi-supervised learning takes place in natural language processing, when you need to solve a task but have few labeled data. In such cases, you need to learn a good representation or language model, so you take sentences and give your network self-supervision tasks like these:

  • Ask the network to predict the next word in a sentence (which you know because you took it away).

  • Mask a word and ask the network to predict which word goes there (which you know because you had to mask it).

  • Change the word for a random one (that probably doesn't make sense) and ask the network which word is wrong.

As you can see, these tasks are fairly simple to formulate and the labels are part of the same sample, but they require a certain understanding of the context to be solved.

And it's always like this: alter your data in some way, generating the label in the process, and ask the model something related to that transformation. If the task requires enough understanding of the data, you'll have success.

| improve this answer | |
  • $\begingroup$ Isn't what you describe "distant learning"? $\endgroup$ – Make42 Jun 25 at 12:04
  • $\begingroup$ I haven't heard of that concept. Could you elaborate on that? $\endgroup$ – David Jul 3 at 8:31
  • $\begingroup$ Sure, I described it in ai.stackexchange.com/questions/22186/…: Distant supervision is a type of weak supervision (there is uncertainty in the labeling) that uses an auxiliary automatic mechanism to produce weak labels / reference output (in contrast to non-expert human labelers) from data. Possibly (I am not clear on that point yet), the data which is used by this mechanism, cannot be from the input data of the main model. $\endgroup$ – Make42 Jul 3 at 9:25
  • $\begingroup$ I see. I'd say the main difference is on the target task. Distant supervision keeps focusing on the same original task, while self-supervision focuses on a surrogate task as a proxy for learning a good representation. I will elaborate this as an answer in that question. $\endgroup$ – David Jul 4 at 11:24
  • $\begingroup$ But, in that other answer you wrote that it might be that "the task casually matches the target task". Then there would be no surrogate task anymore and if that is still self-supervision, then having a surrogate task is not a requirement for a learning to be self-supervised. Aren't you contradicting yourself...? $\endgroup$ – Make42 Jul 4 at 12:06

Self-supervised visual recognition is often applied to representation learning. Here we first learn features on unlabeled data (representation learning), and then learn the real model on features extracted from the labeled data. This especially makes sense when we have a lot of unlabeled data and few labeled data.

The features can be learned by solving so called pretext tasks. Examples of pretext tasks are to predict rotation of a jittered image, to recognize jittered instances of a same image, or to predict spatial relationship of image patches.

A nice overview and interesting results can be found in this recent paper.

| improve this answer | |
  • $\begingroup$ Do you call the entire setup "self-supervised learning" or only the representation learning part? What is the difference between the terms "self-supervised learning" and "representation learning" then? $\endgroup$ – Make42 Jun 25 at 12:08
  • $\begingroup$ Self-supervised learning is a way to achieve representation learning. Some other ways to achieve the same goal are supervised learning and unsupervised learning. $\endgroup$ – ssegvic Jun 26 at 13:05
  • $\begingroup$ Ok, I understand you. However, this is in contrast to the answer of @nbro who does not mention the solving of an auxiliary task, implying that an auxiliary task is not required for calling something "self-supervised". See my question ai.stackexchange.com/questions/22184/… $\endgroup$ – Make42 Jun 26 at 13:13
  • $\begingroup$ @Make42 I've updated my answer to explicitly mention the concept of pretext or auxiliary task. In the specific context of the robotics paper, which my answer is particularly based on, the concept of pretext or auxilary task is not made explicit, but these terms are used in other contexts, so it's important to emphasize them. $\endgroup$ – nbro Jun 26 at 18:32

Not the answer you're looking for? Browse other questions tagged or ask your own question.