2
$\begingroup$

Recently, I have seen a surge of papers w.r.t contrastive learning (a subset of semi-supervised learning).

Can anyone give a detailed explanation of this approach with its advantages/disadvantages and what are the cases in which it gives better results?

Also, why it's gaining traction amongst the ML research community?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ As I understand it contrastive learning is a type of data augmentation. For classification you have labeled dataset, two augmentation of the same sample should have the same label. Idea of contrastive learning: not only two different augmentation of the same sample should have the same label, but the label should be different from the sample augmented from different label. Contrastive learning use loss which encourage that difference (therefore "contrastive") Advantages and disadvantages are the same as for common augmentation. Based on arxiv.org/abs/2002.05709 $\endgroup$
    – mirror2image
    Mar 8, 2020 at 15:15

1 Answer 1

Reset to default
1
$\begingroup$

Contrastive learning is a framework that learns similar/dissimilar representations from data that are organized into similar/dissimilar pairs. This can be formulated as a dictionary look-up problem.

If I conceptually compare the loss mechanisms for:

Both MoCo and SimCLR use varients of a contrastive loss function, like InfoNCE from the paper Representation Learning with Contrastive Predictive Coding \begin{eqnarray*} \mathcal{L}_{q,k^+,\{k^-\}}=-log\frac{exp(q\cdot k^+/\tau)}{exp(q\cdot k^+/\tau)+\sum\limits_{k^-}exp(q\cdot k^-/\tau)} \end{eqnarray*}

Here q is a query representation, $k^+$ is a representation of the positive (similar) key sample, and ${k^−}$ are representations of the negative (dissimilar) key samples. $\tau$ is a temperature hyper-parameter. In the instance discrimination pretext task (used by MoCo and SimCLR), a query and a key form a positive pair if they are data-augmented versions of the same image, and otherwise form a negative pair. SimCLR vs MoCo

The contrastive loss can be minimized by various mechanisms that differ in how the keys are maintained.

In an end-to-end mechanism (Fig. 1a), the negative keys are from the same batch and updated end-to-end by back-propagation. SimCLR, is based on this mechanism and requires a large batch to provide a large set of negatives.

In the MoCo mechanism i.e. Momentum Contrast (Fig. 1b), the negative keys are maintained in a queue, and only the queries and positive keys are encoded in each training batch.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .