0
$\begingroup$

I am a researcher in a field, and new to the whole of AI and machine learning techniques. May the following question is trivial or not framed in the ML language but I try my best.

I have two sets of representations (I can extract feature vectors, etc., from the datasets) from vastly different domains. I want to find, if any, a relationship exists between these two sets. In other words, I want an algorithm (the idea of an algorithm) to learn both representations and find the connections and convert one representation to another.

There is neither apparent one-to-one correspondence nor both need to be the same lengths.

Any suggestion on how to approach this problem is appreciated.

I thought of one method; write an encoder-decoder for each of these presentations separately and swap the decoders. I am not sure whether it works or not, and besides I may not have any idea what's going on there.

I prefer a general approach if it exists.

Improve this question
$\endgroup$
3
  • $\begingroup$ Have you tried a simple design where you input one representation to an NN and make it predict the other? $\endgroup$
    – user253751
    Sep 18, 2021 at 18:59
  • $\begingroup$ You say "There is neither apparent one-to-one correspondence nor both need to be the same lengths", but then I don't understand why you want to convert one representation to the other. What's your ultimate goal or why do you want to do this? Could you also provide an example of 2 vastly different feature vectors that you would want the algorithm to convert from/to, so that we have a better idea of what you really want to achieve? $\endgroup$
    – nbro
    Sep 19, 2021 at 0:43
  • $\begingroup$ Yes, I am sorry that the message was kind of cryptic. I am comparing chemical vibrations to musical notes. I have a dataset of chemicals, let's say proteins, polymers, crystals etc. and another dataset of musical notes, ragas etc. I need to find the correspondence or any hidden structural similarities between these two domains. I can assign one amino acid to one note manually and create some kind of music but that's not what I want. In the end, the algorithm needs to convert chemicals to music or music to chemicals vice versa. $\endgroup$
    – bababee
    Sep 30, 2021 at 22:01

1 Answer 1

Reset to default
1
$\begingroup$

Well, I suppose one can use some kind of contrastive learning in this case.

A famous example of the establishment of relation between two different representations is the CLIP - Contrastive Language–Image Pre-training, where model gets a huge corpus of image captions and images and the image caption is passed through the language model, and the image itself through convolutional or ViT backbone, and the model learns to make the image embedding (output of the visual model) to be similar to the output embedding (output of text model) and dissimilar (in the sense of cosine distance) to the embedding of captions, that belong to other images in the training dataset.

enter image description here

In case your two data representations allow for the notion of similar and dissimilar, I expect you can apply a similar procedure.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you. It is very interesting. $\endgroup$
    – bababee
    Sep 18, 2021 at 21:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .