Does self-supervised learning require auxiliary tasks?

Question

Self-supervised learning algorithms provide labels automatically. But, it is not clear what else is required for an algorithm to fall under the category "self-supervised":

Some say, self-supervised learning algorithms learn on a set of auxiliary tasks [1], also named pretext task [2, 3], instead of the task we are interested in. Further examples are word2vec or autoencoders [4] or word2vec [5]. Here it is sometimes mentioned that the goal is to "expose the inner structure of the data".

Others do not mention that, implying that some algorithms can be called to be "self-supervised learning algorithms" if they are directly learning the task we are interested in [6, 7].

Is the "auxiliary tasks" a requirement for a training setup to be called "self-supervised learning" or is it just optional?

---

Research articles mentioning the auxiliary / pretext task:

1. Revisiting Self-Supervised Visual Representation Learning, 2019, mentioned by [3]:

The self-supervised learning framework requires only unlabeled data in order to formulate a pretext learning task such as predicting context or image rotation, for which a target objective can be computed without supervision.

2. Unsupervised Representation Learning by Predicting Image Rotations, ICLR, 2018, mentioned by [2]:

a prominent paradigm is the so-called self-supervised learning that defines an annotation free pretext task, using only the visual information present on the images or videos, in order to provide a surrogatesupervision signal for feature learning.

3. Unsupervised Visual Representation Learning by Context Prediction, 2016, mentioned by [2]:

This converts an apparently unsupervised problem (finding a good similarity metric between words) intoa “self-supervised” one: learning a function from a givenword to the words surrounding it. Here the context predic-tion task is just a “pretext” to force the model to learn agood word embedding, which, in turn, has been shown tobe useful in a number of real tasks, such as semantic wordsimilarity.

4. Scaling and Benchmarking Self-Supervised Visual Representation Learning, 2019:

In discriminative self-supervised learning, which is the main focus of this work, a model is trained on an auxiliary or ‘pretext’ task for which ground-truth is available for free. In most cases, the pretext task involves predicting some hidden portion of the data (for example, predicting color for gray-scale images

Answer 1

Most (if not all) self-supervised learning techniques for (visual or textual) representation learning use pre-text tasks, and many pre-text tasks have been proposed in recent years.

However, as I say in my other answer (which you cite), the term SSL has also been used (at least, in robotics: for example, see this paper, which I am very familiar with) to refer to techniques that automatically (although approximately) label the unlabelled dataset for your downstream task (i.e. image recognition), i.e. they automatically create a labeled dataset of pairs $(x_i, \hat{y}_i)$, where $x_i$ is an image that contains an object and $\hat{y}_i$ is the automatically (and possibly approximately) generated label (such as "dog"). This latter use of the term SSL is closer to some weakly supervised learning (WSL) techniques. Actually, it can be considered a WSL technique.

Now, in this specific paper, they actually solve some kind of pre-text task, i.e. they exploit the relations between two different sensors to produce the labels.

To answer your question more directly: in all SSL papers that I have come across, some kind of pre-text task is always solved, i.e., in some way, you need to automatically generate the supervisory signal, and that task that we solve with the automatically generated learning signal (with the purpose of learning representations or generating a labeled dataset) can be considered the pre-text task (which may coincide with the downstream task, for example, in the case you're training an auto-encoder with an unlabelled dataset for the purposes of image denoising).

In any case, I wouldn't bother too much about it. Just keep your context in mind when reading your paper. If you're really worried about it, then you should probably read almost all SSL-related papers, but, in that case, by the end of that, you will be an expert on the topic and you will not need our help (or my help).