2
$\begingroup$

Weak supervision is supervised learning, with uncertainty in the labeling, e.g. due to automatic labeling or because non-experts labelled the data [1].

Distant supervision [2, 3] is a type of weak supervision that uses an auxiliary automatic mechanism to produce weak labels / reference output (in contrast to non-expert human labelers).

According to this answer

Self-supervised learning (or self-supervision) is a supervised learning technique where the training data is automatically labelled.

In the examples for self-supervised learning, I have seen so far, the labels were extracted from the input data.

What is the difference between distant supervision and self-supervision?

(Setup mentioned in discussion:

enter image description here

Improve this question
$\endgroup$
3
  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$
    – nbro
    Jun 28, 2020 at 15:34
  • $\begingroup$ I have updated my answer. Anyway, after having read something about distant supervision, it seems like it refers to learning where the data comes from a remote database rather than a local dataset (and so the term "distant supervision" makes sense). In any case, this term is definitely less common than SSL. $\endgroup$
    – nbro
    Aug 4, 2020 at 18:03
  • $\begingroup$ @nbro: Which articles have you read regarding distant supervision? Maybe I can also have a look. $\endgroup$
    – Make42
    Aug 5, 2020 at 8:26

1 Answer 1

Reset to default
1
$\begingroup$

The main difference between distant supervision (as described in the link you provided) and self-supervision lies on the task the network is trained on.

Distant supervision focuses on generating weak labels for the very same task that would be tackled with supervised labels, and the final result could be directly used for that matter.

Self-supervision is a means for learning a data representation. It does so by learning a surrogate task, which is defined by inputs and labels derived exclusively from the original input data.

I can imagine cases in which an implementation of self-supervision could be considered distant supervision (if the task casually matches the target task). On the other hand, if external data sources would be employed for training on a surrogate task, that would be a case of representation learning (that could incorporate self-supervision too).

Improve this answer
$\endgroup$
7
  • $\begingroup$ Ok, taking ai.stackexchange.com/questions/10623/… into account, the surrogate task is not the requirement, but learning from the input data is. While distant learning seems to be about learning from external data. I got that much. Now, the example of ai.stackexchange.com/a/10624/38174 would then not be self-supervised... unless you say that the proximity sensor is part of the "input", but then anything could be "part of the input". $\endgroup$
    – Make42
    Jul 4, 2020 at 12:16
  • $\begingroup$ I would say the input must be the input of the target class, or not? But then, with a surrogate task, the "input" where the labels are extracted from, is not the input of the target class. E.g. the documents of word2vec would be the input where the labels are extracted from, but the input of a target task is the (one-hot) word-vector. So "input of target task" has its origin of the dataset from which labels are extracted, but it is not the same. Or am I missing something? In variational autoencoder they aren't the same either. In "normal" autoencoders however they are the same. $\endgroup$
    – Make42
    Jul 4, 2020 at 12:18
  • $\begingroup$ Also, a lot of the publications mentioned in ai.stackexchange.com/questions/22184/… claim that the auxiliary task the deciding factor. (The more I look into the naming in the machine learning literature, the more frustrating it gets.) $\endgroup$
    – Make42
    Jul 4, 2020 at 12:21
  • $\begingroup$ 1) You got it right in the first comment. The robotics example is a bit farfetched, but it's correct as far as the sensor data is part of the input, but only trustworthy in some cases. 2) The input is usually modified in order to generate the auxiliary task e.g. masking a word in order to predict it afterwards. 3) The auxiliary task is key due to its implications in the representation learned e.g. predicting the mean color of an image is probably a very poor choice if you aim to perform facial recognition. $\endgroup$
    – David
    Jul 4, 2020 at 12:36
  • $\begingroup$ 1) When we say that "self-supervision requires that data labels are extracted from the input data", we need to define what the input data is. Look at the new graphic in the question: Is it the input to the target task (3), the input to the aux. task (2) or a data that lies even further "outside" (1). For word2vec it is (1). For variational autoencoders used for image denoising and my similar task we are actually extracting the (3) from the (1) and the off-the-shelf-data is the out target data. But: if (1) is the input data for the definition then anything could be part of that input data. $\endgroup$
    – Make42
    Jul 4, 2020 at 15:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .