21 votes
Accepted

What is the difference between latent and embedding spaces?

Embedding vs Latent Space Due to Machine Learning's recent and rapid renaissance, and the fact that it draws from many distinct areas of mathematics, statistics, and computer science, it often has a ...
user avatar
5 votes
Accepted

Is an embedding a representation of a word or its meaning?

An embedding is a representation of a word that can be used as a proxy for some of its linguistic properties. The 'human' representation of a word, a sequence of letters and other symbols, is not ...
user avatar
  • 5,062
4 votes
Accepted

What are knowledge graph embeddings?

Knowledge graph embeddings (KGE) are embeddings created in the context of a knowledge graph (KG), which can be viewed as a visual/graphical representation of a knowledge base, where nodes are entities ...
user avatar
  • 34.4k
4 votes

What is the intuition behind how word embeddings bring information to a neural network?

Shakespeare once said "A rose by any other name would smell as sweet" (Romeo and Juliet). Words are just labels we attach to ideas for convenience. By using one hot we remain tied to the letter ...
user avatar
4 votes

What is the difference between latent and embedding spaces?

The expression "latent space" explicitly indicates that the space is associated with the mathematical concept of an hidden (or latent) variable, which cannot be observed directly, but only indirectly. ...
user avatar
  • 34.4k
3 votes
Accepted

Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram?

The reason most music-generation models use discrete representations is because the long-term structures of music are very challenging to model. Note that the MIDI data in MAESTRO (used in the two ...
user avatar
  • 194
2 votes

What is the intuition behind how word embeddings bring information to a neural network?

Adding to Colin's answer; using word embedding tend to be much more robust that one-hot vectors. Consider the the following two sentences: The desk has a book on it. and The table has a book on ...
user avatar
  • 316
2 votes
Accepted

Converting age and sex variables to a 64-unit dense layer

Convert them into numbers (using one-hot vectors or direct numerical representations) and then concatenate them. Then, you can pass them through the Dense layer.
user avatar
2 votes

How does the embeddings work in vision transformer from paper?

In Machine Learning "embedding" means taking some set of raw inputs (like natural language tokens in NLP or image patches in your example) and converting them to vectors somehow. The ...
user avatar
  • 1,833
2 votes
Accepted

What is the role of embeddings in a deep recurrent Q network?

The purpose of the input network is to embed the input tuple into a state/task representation, that can then be fed into the RNN hidden state at each time step. $(o^a_t,m^a′_{t−1},u^a_{t−1},a)$ (input)...
user avatar
  • 517
2 votes

Is an embedding a representation of a word or its meaning?

Although we have had multiple similar questions (see here, here and here) and it seems to me that you focused on word embeddings (probably because you were not aware of the application of embeddings ...
user avatar
  • 34.4k
2 votes

How to determine the embedding size?

In most cases, seems that embedding dim is chosen empirically, by trial and error. Older papers in NLP used 300 conventionally https://petuum.medium.com/embeddings-a-matrix-of-meaning-4de877c9aa27. ...
user avatar
2 votes

Sequence Embedding using embedding layer: how does the network architecture influence it?

Premises: mine is not gonna be an exhaustive answer, also I'm more familiar with classic natural language processing than with embedding vectors applied to protein sequences. Said so, I think I can ...
user avatar
1 vote

What is the distribution of autoencoder embeddings?

If you don't make assumptions about your input distribution, and the form of your network, it's very difficult to express the embedding distribution in closed-form. Here's an idea on how to formalize ...
user avatar
1 vote

Why are embeddings added, not concatenated?

First of all, I think it is very hard to properly reason about these things, but there are a few points that might justify using sum instead of concatenation. For example, concatenation would have the ...
user avatar
  • 671
1 vote
Accepted

Is "node embedding" in GNN analogous to "hidden layer" of FFN?

Embeddings are vectors. Layers are functions. So, node embeddings (e.g. produced by TransE) are analogous to word embeddings or code embeddings, i.e. they are vector (and lower-dimensional) ...
user avatar
  • 34.4k
1 vote

Sensible integer embedding/encoding for distinguishing elements of a set?

As you have asserted that the id is not meaningful of itself, it probably doesn't matter how you encode it. I would recommend the following, in order: Don't encode the id. It is not clear that you ...
user avatar
  • 24.5k
1 vote

How to determine the embedding size?

I get an answer from this book: Machine Learning Design Patterns: Solutions to Common Challenges in Data Preparation, Model Building, and MLOps. If we’re in a hurry, one rule of thumb is to use the ...
user avatar
1 vote

Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram?

These papers are also very close to what I meant in the question (too long for a comment). The following references come mostly from work on speech recognition. Mockingjay In this work, they use an ...
user avatar
1 vote

What is the difference between latent and embedding spaces?

To give a statistician's answer, the distinction is empirical (embedding) versus theoretical (latent positions). You define a statistical model which has latent positions that you could then try to ...
user avatar
  • 11
1 vote
Accepted

In the machine learning literature, what does it mean to say that something is "embedded" in some space?

Embedding is the process of representing data (from a source domain) in a new (or target) domain. Usually, the source domain is discrete, and the target domain is continuous. For example, embedding ...
user avatar
  • 1,663

Only top scored, non community-wiki answers of a minimum length are eligible