# What is self-supervised learning in machine learning?

What is self-supervised learning in machine learning? How is it different from supervised learning?

• Comments are not for extended discussion; this conversation has been moved to chat. – nbro Mar 6 at 1:12

Self-supervised learning (or self-supervision) is a relatively recent learning technique (in machine learning) where the training data is autonomously (or automatically) labelled. It is still supervised learning, but the datasets do not need to be manually labelled by a human, but they can e.g. be labelled by finding and exploiting the relations (or correlations) between different input signals (that is, input coming from different sensor modalities).

A natural advantage and consequence of self-supervised learning is that it can more easily (with respect to e.g. supervised learning) be performed in an online fashion (given that data can be gathered and labelled without human intervention), where models can be updated or trained completely from scratch. Therefore, self-supervised learning should also be well suited for changing environments, data and, in general, situations.

For example, consider a robot which is equipped with a proximity sensor (which is a short-range sensor capable of detecting objects in front of the robot at short distances) and a camera (which is long-range sensor, but which does not provide a direct way of detecting objects). You can also assume that this robot is capable of performing odometry. An example of such a robot is Mighty Thymio.

Consider now the task of detecting objects in front of the robot at longer ranges than the range the proximity sensor allows. In general, we could train a CNN to achieve that. However, to train such CNN, in supervised learning, we would first need a labelled dataset, which contains labelled images (or videos), where the labels could e.g. be "object in the image" or "no object in the image". In supervised learning, this dataset would need to be manually labelled by a human, which clearly would require a lot of work.

To overcome this issue, we can use a self-supervised learning approach. In this example, the basic idea is to associate the output of the proximity sensors at a time step $$t' > t$$ with the output of the camera at time step $$t$$ (a smaller time step than $$t'$$).

More specifically, suppose that the robot is initially at coordinates $$(x, y)$$ (on the plane), at time step $$t$$. At this point, we still do not have enough info to label the output of the camera (at the same time step $$t$$). Suppose now that, at time $$t'$$, the robot is at position $$(x', y')$$. At time step $$t'$$, the output of the proximity sensor will e.g. be "object in front of the robot" or "no object in front of the robot". Without loss of generality, suppose that the output of the proximity sensor at $$t' > t$$ is "no object in front of the robot", then the label associated with the output of the camera (an image frame) at time $$t$$ will be "no object in front of the robot".

For more details about this specific example, have a look at the paper Learning Long-range Perception using Self-Supervision from Short-Range Sensors and Odometry by Mirko Nava, Jérôme Guzzi, R. Omar Chavez-Garcia, Luca M. Gambardella and Alessandro Giusti.

Note that self-supervised learning is defined slightly differently depending on the context or area, which can, for example, be robotics, reinforcement learning or representation (or feature) learning. More precisely, the definition given above is used in robotics. See, for example, also this paper Multi-task Self-Supervised Visual Learning. For a slightly different definition of self-supervised learning, see, for example, the paper Digging Into Self-Supervised Monocular Depth Estimation.

For another introduction to self-supervised learning, have a look at this web article: https://hackernoon.com/self-supervised-learning-gets-us-closer-to-autonomous-learning-be77e6c86b5a. In this article, the author also compares self-supervised learning to unsupervised learning, semi-supervised learning and reinforcement learning.

Furthermore, if you want to know current methods being used for self-supervised learning, take a look at this article: https://amitness.com/2020/02/illustrated-self-supervised-learning . In this article, the author summarizes key ideas of existing self-supervised methods with diagrams and visualization.

There is also a curated list of links to papers where this learning approach is used at the following URL: https://github.com/jason718/awesome-self-supervised-learning.

• I disagree that self-supervised learning means that the data is labeled, such that the output layer should ideally equal the layers. Take this and this work for example, they both say that they use self-supervised learning, but neither uses any labels. Instead, the loss is calculated by warping a frame in the video according to the output of the network and comparing the warped frame with another frame. – HelloGoodbye Jul 3 '19 at 22:22
• @HelloGoodbye There are different definitions of self-supervised learning, indeed. I should update this answer. The definition I used is the most common one and it used in robotics. All other ones are not very related to supervised learning, so their usage is a little bit inappropriate or misleading. This usage of self-supervised learning, in robotics, is the most appropriate, given its relation to supervised learning. – nbro Jul 3 '19 at 22:40
• Thanks for linking my post on hacker noon – Dickey Singh Aug 2 '19 at 3:06

Self-supervised learning is when you use some parts of the samples as labels for a task that requires a good degree of comprehension to be solved. I'll emphasize these two key points, before giving an example:

• Labels are extracted from the sample, so they can be generated automatically, with some very simple algorithm (maybe just random selection).

• The task requires understanding. This means that, in order to predict the output, the model has to extract some good patterns from the data, generating on the process a good representation.

A very common case for semi-supervised learning takes place in natural language processing, when you need to solve a task but have few labeled data. In such cases, you need to learn a good representation or language model, so you take sentences and give your network self-supervision tasks like these:

• Ask the network to predict the next word in a sentence (which you know because you took it away).

• Mask a word and ask the network to predict which word goes there (which you know because you had to mask it).

• Change the word for a random one (that probably doesn't make sense) and ask the network which word is wrong.

As you can see, these tasks are fairly simple to formulate and the labels are part of the same sample, but they require a certain understanding of the context to be solved.

And it's always like this: alter your data in some way, generating the label in the process, and ask the model something related to that transformation. If the task requires enough understanding of the data, you'll have success.

Self-supervised visual recognition is often applied to representation learning. Here we first learn features on unlabeled data (representation learning), and then learn the real model on features extracted from the labeled data. This especially makes sense when we have a lot of unlabeled data and few labeled data.

The features can be learned by solving so called pretext tasks. Examples of pretext tasks are to predict rotation of a jittered image, to recognize jittered instances of a same image, or to predict spatial relationship of image patches.

A nice overview and interesting results can be found in this recent paper.