0
$\begingroup$

I am specifically interested in data2vec, Meta's new model that can convert image, text, and sound data into a unified neural network representation. To my understanding, they did this through self-supervised learning by masking parts of the input and having the network predict the hidden states if the input hadn't been masked. This allows these modes to share a common representation.

However, I don't understand how the representations of different modes can be connected. For example, how are the hidden state representations of an image of a banana and the word banana trained to be similar, if they are at all?

Improve this question
$\endgroup$
1
  • $\begingroup$ One option would be an association network between the various individual latent spaces, having its own multi-modal latent space, where the shared phenomena reside. $\endgroup$
    – Michael
    Mar 21, 2022 at 9:10

2 Answers 2

Reset to default
0
$\begingroup$

I agree with the answer by @Funzo. The corresponding section that clarifies this in the paper can be found on page three:

Multimodal pre-training. [...] Our work does not perform multimodal training but aims to unifiy the learning objective for self-supervised learning in different modalities. We hope that this will enable better multimodal representations in the future. [ArXiv]

So what the authors contribute is a framework for learning rich data representations, given any type of data. The framework is the training pipeline that uses the same network in a teacher and student setup. The network first generates data representations from unmasked data (teacher mode) and then does the same with a masked version of the data (student mode). The training objective is minimizing the error of all latent representations between the network in student mode and in teacher mode. If you were to apply the approach you have to choose a model suitable for your data (e.g. the authors use a Vision Transformer for Images).

Improve this answer
$\endgroup$
-1
$\begingroup$

it's not multimodal. It's trying to be a standard way of training a for a model. But it is not suppose to work across modalities

Improve this answer
$\endgroup$
1
  • $\begingroup$ Hi @Funzo, welcome to AI.SE. Do you think you could perhaps expand this answer a bit? As it stands, it makes some claims, but doesn't explain them or provide any sources. Adding sources and explaining your answer some more would greatly improve the quality of the post. Thank you! $\endgroup$
    – Mithical ♦
    Jun 2, 2022 at 3:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .