# Tag Info

## Hot answers tagged embeddings

21 votes
Accepted

### What is the difference between latent and embedding spaces?

Embedding vs Latent Space Due to Machine Learning's recent and rapid renaissance, and the fact that it draws from many distinct areas of mathematics, statistics, and computer science, it often has a ...
• 801
5 votes
Accepted

### Is an embedding a representation of a word or its meaning?

An embedding is a representation of a word that can be used as a proxy for some of its linguistic properties. The 'human' representation of a word, a sequence of letters and other symbols, is not ...
• 5,062
4 votes
Accepted

### What are knowledge graph embeddings?

Knowledge graph embeddings (KGE) are embeddings created in the context of a knowledge graph (KG), which can be viewed as a visual/graphical representation of a knowledge base, where nodes are entities ...
• 34.4k
4 votes

### What is the intuition behind how word embeddings bring information to a neural network?

Shakespeare once said "A rose by any other name would smell as sweet" (Romeo and Juliet). Words are just labels we attach to ideas for convenience. By using one hot we remain tied to the letter ...
4 votes

### What is the difference between latent and embedding spaces?

The expression "latent space" explicitly indicates that the space is associated with the mathematical concept of an hidden (or latent) variable, which cannot be observed directly, but only indirectly. ...
• 34.4k
3 votes
Accepted

### Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram?

The reason most music-generation models use discrete representations is because the long-term structures of music are very challenging to model. Note that the MIDI data in MAESTRO (used in the two ...
• 194
2 votes

### What is the intuition behind how word embeddings bring information to a neural network?

Adding to Colin's answer; using word embedding tend to be much more robust that one-hot vectors. Consider the the following two sentences: The desk has a book on it. and The table has a book on ...
• 316
2 votes
Accepted

### Converting age and sex variables to a 64-unit dense layer

Convert them into numbers (using one-hot vectors or direct numerical representations) and then concatenate them. Then, you can pass them through the Dense layer.
2 votes

### How does the embeddings work in vision transformer from paper?

In Machine Learning "embedding" means taking some set of raw inputs (like natural language tokens in NLP or image patches in your example) and converting them to vectors somehow. The ...
• 1,833
2 votes
Accepted

### What is the role of embeddings in a deep recurrent Q network?

The purpose of the input network is to embed the input tuple into a state/task representation, that can then be fed into the RNN hidden state at each time step. $(o^a_t,m^a′_{t−1},u^a_{t−1},a)$ (input)...
• 517
2 votes

### Is an embedding a representation of a word or its meaning?

Although we have had multiple similar questions (see here, here and here) and it seems to me that you focused on word embeddings (probably because you were not aware of the application of embeddings ...
• 34.4k
2 votes

### How to determine the embedding size?

In most cases, seems that embedding dim is chosen empirically, by trial and error. Older papers in NLP used 300 conventionally https://petuum.medium.com/embeddings-a-matrix-of-meaning-4de877c9aa27. ...
2 votes

### Sequence Embedding using embedding layer: how does the network architecture influence it?

Premises: mine is not gonna be an exhaustive answer, also I'm more familiar with classic natural language processing than with embedding vectors applied to protein sequences. Said so, I think I can ...
• 3,983
1 vote

### What is the distribution of autoencoder embeddings?

If you don't make assumptions about your input distribution, and the form of your network, it's very difficult to express the embedding distribution in closed-form. Here's an idea on how to formalize ...
1 vote

### Why are embeddings added, not concatenated?

First of all, I think it is very hard to properly reason about these things, but there are a few points that might justify using sum instead of concatenation. For example, concatenation would have the ...
• 671
1 vote
Accepted

### Is "node embedding" in GNN analogous to "hidden layer" of FFN?

Embeddings are vectors. Layers are functions. So, node embeddings (e.g. produced by TransE) are analogous to word embeddings or code embeddings, i.e. they are vector (and lower-dimensional) ...
• 34.4k
1 vote

### Sensible integer embedding/encoding for distinguishing elements of a set?

As you have asserted that the id is not meaningful of itself, it probably doesn't matter how you encode it. I would recommend the following, in order: Don't encode the id. It is not clear that you ...
• 24.5k
1 vote

### How to determine the embedding size?

I get an answer from this book: Machine Learning Design Patterns: Solutions to Common Challenges in Data Preparation, Model Building, and MLOps. If we’re in a hurry, one rule of thumb is to use the ...
• 747
1 vote

### Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram?

These papers are also very close to what I meant in the question (too long for a comment). The following references come mostly from work on speech recognition. Mockingjay In this work, they use an ...
1 vote

### What is the difference between latent and embedding spaces?

To give a statistician's answer, the distinction is empirical (embedding) versus theoretical (latent positions). You define a statistical model which has latent positions that you could then try to ...
• 11
1 vote
Accepted

### In the machine learning literature, what does it mean to say that something is "embedded" in some space?

Embedding is the process of representing data (from a source domain) in a new (or target) domain. Usually, the source domain is discrete, and the target domain is continuous. For example, embedding ...
• 1,663

Only top scored, non community-wiki answers of a minimum length are eligible