Facebook has just pushed out a bigger version of their multi-lingual language model XLM, called XLM-R. My question is: do these kind of multi-lingual models imply, or even ensure, that their embeddings are comparable between languages? That is, are semantically related words close together in the vector space across languages?
Perhaps the most interesting citation from the paper that is relevant to my question (p. 3):
Unlike Lample and Conneau (2019), we do not use language embeddings, which allows our model to better deal with code-switching.
Because they do not seem to make a distinction between languages, and there's just one vocabulary for all trained data, I fail to see how this can be truly representative of semantics anymore. The move away from semantics is increased further by the use of BPE, since morphological features (or just plain, statistical word chunks) of one language might often not be semantically related to the same chunk in another language - this can be true for tokens themselves, but especially so for subword information.
So, in short: how well can the embeddings in multi-lingual language models be used for semantically comparing input (e.g. a word or sentence) of two different languages?