12

Embedding vs Latent Space Due to Machine Learning's recent and rapid renaissance, and the fact that it draws from many distinct areas of mathematics, statistics, and computer science, it often has a number of different terms for the same or similar concepts. "Latent space" and "embedding" both refer to an (often lower-dimensional) ...


5

An embedding is a representation of a word that can be used as a proxy for some of its linguistic properties. The 'human' representation of a word, a sequence of letters and other symbols, is not related at all to its meaning or use in actual text. It only serves as a look-up key into our cognitive language processing facility (however that actually works) ...


3

Shakespeare once said "A rose by any other name would smell as sweet" (Romeo and Juliet). Words are just labels we attach to ideas for convenience. By using one hot we remain tied to the letter sequence r,o,s,e, and some other structure must take on the responsibility of attaching the context of sweetness to it. Word embeddings learn a multi-dimensional ...


3

The expression "latent space" explicitly indicates that the space is associated with the mathematical concept of an hidden (or latent) variable, which cannot be observed directly, but only indirectly. The expression "embedding space" refers to a vector space that represents an original space of inputs (e.g. images or words). For example, in the case of "...


3

The reason most music-generation models use discrete representations is because the long-term structures of music are very challenging to model. Note that the MIDI data in MAESTRO (used in the two papers you linked) encodes performances, not scores, so they include timing and accents of real performers--but are still sequences of discrete events, not audio. ...


2

The purpose of the input network is to embed the input tuple into a state/task representation, that can then be fed into the RNN hidden state at each time step. $(o^a_t,m^a′_{t−1},u^a_{t−1},a)$ (input) $\rightarrow$ input network (embedding) $\rightarrow$ $z_t$ (task representation) According to to section 6.1 of the paper, the input is a tuple represented ...


2

Convert them into numbers (using one-hot vectors or direct numerical representations) and then concatenate them. Then, you can pass them through the Dense layer.


2

Although we have had multiple similar questions (see here, here and here) and it seems to me that you focused on word embeddings (probably because you were not aware of the application of embeddings to other contexts), in addition to what is stated in the other answer, it's important to note that the concept of an embedding does not just apply to words. For ...


2

In most cases, seems that embedding dim is chosen empirically, by trial and error. Older papers in NLP used 300 conventionally https://petuum.medium.com/embeddings-a-matrix-of-meaning-4de877c9aa27. More recent papers used 512, 768, 1024. One of the factors, influencing the choice of embedding is the way you would like different vectors to correlate with each ...


1

I get an answer from this book: Machine Learning Design Patterns: Solutions to Common Challenges in Data Preparation, Model Building, and MLOps. If we’re in a hurry, one rule of thumb is to use the fourth root of the total number of unique categorical elements while another is that the embedding dimension should be approximately 1.6 times the square root of ...


1

These papers are also very close to what I meant in the question (too long for a comment). The following references come mostly from work on speech recognition. Mockingjay In this work, they use an analogy of Bert architecture that is fed by Mel-spectrogram, with some audio segments "masked". The model is asked to reconstruct the masked parts. To ...


1

Embedding is the process of representing data (from a source domain) in a new (or target) domain. Usually, the source domain is discrete, and the target domain is continuous. For example, embedding words into the continuous vector space can be done by the word2vec method. The main reason behind using the embedding is doing meaningful mathematical ...


1

To give a statistician's answer, the distinction is empirical (embedding) versus theoretical (latent positions). You define a statistical model which has latent positions that you could then try to estimate, given data. Or, given data, you might simply find a vector representation of each object of interest in a way that makes sense for the applications ...


1

Adding to Colin's answer; using word embedding tend to be much more robust that one-hot vectors. Consider the the following two sentences: The desk has a book on it. and The table has a book on it. These two sentences are almost identical in meaning. If we were to using word embeddings, the vectors 'desk' and 'table' would be very close together. The ...


Only top voted, non community-wiki answers of a minimum length are eligible