37 votes
Accepted

What is the difference between latent and embedding spaces?

Embedding vs Latent Space Due to Machine Learning's recent and rapid renaissance, and the fact that it draws from many distinct areas of mathematics, statistics, and computer science, it often has a ...
brazofuerte's user avatar
  • 1,031
6 votes

Why are embeddings added, not concatenated?

In high-dimensional spaces, the token embeddings and positional encodings can be thought of as forming two separate subspaces that are approximately orthogonal to each other. This is based on the ...
Abdur Rahman's user avatar
5 votes

How to determine the embedding size?

In most cases, seems that embedding dim is chosen empirically, by trial and error. Older papers in NLP used 300 conventionally https://petuum.medium.com/embeddings-a-matrix-of-meaning-4de877c9aa27. ...
spiridon_the_sun_rotator's user avatar
5 votes
Accepted

Is an embedding a representation of a word or its meaning?

An embedding is a representation of a word that can be used as a proxy for some of its linguistic properties. The 'human' representation of a word, a sequence of letters and other symbols, is not ...
Oliver Mason's user avatar
  • 5,397
5 votes

What is the difference between latent and embedding spaces?

The expression "latent space" explicitly indicates that the space is associated with the mathematical concept of an hidden (or latent) variable, which cannot be observed directly, but only indirectly. ...
nbro's user avatar
  • 40.1k
5 votes

What is the intuition behind how word embeddings bring information to a neural network?

Shakespeare once said "A rose by any other name would smell as sweet" (Romeo and Juliet). Words are just labels we attach to ideas for convenience. By using one hot we remain tied to the letter ...
Colin Beckingham's user avatar
4 votes

How to determine the embedding size?

There is a rule of thumb that says min(50, num_categories/2). But this tops out at 100 categories, what to do after that? I propose this: When num_categories <= 1000: ...
aboveandbeyondis's user avatar
4 votes

Creating a support chat bot for my business

First and foremost, do not use GPT/OpenAI for customer-facing applications. You end up with a mess. GPT is great for creative work, but not for production. GPT is a probabilistic language model, and ...
kokumajutsu's user avatar
3 votes
Accepted

What is the difference between representation and embedding?

Vector representation is a generic term used to talk about any type of feature encoding, embedding vectors are instead a special case of vector representation. When talking about vector representation ...
Edoardo Guerriero's user avatar
3 votes

Sequence Embedding using embedding layer: how does the network architecture influence it?

Premises: mine is not gonna be an exhaustive answer, also I'm more familiar with classic natural language processing than with embedding vectors applied to protein sequences. Said so, I think I can ...
Edoardo Guerriero's user avatar
3 votes
Accepted

What are knowledge graph embeddings?

Knowledge graph embeddings (KGE) are embeddings created in the context of a knowledge graph (KG), which can be viewed as a visual/graphical representation of a knowledge base, where nodes are entities ...
nbro's user avatar
  • 40.1k
3 votes

What is the intuition behind how word embeddings bring information to a neural network?

Adding to Colin's answer; using word embedding tend to be much more robust that one-hot vectors. Consider the the following two sentences: The desk has a book on it. and The table has a book on ...
Daniel's user avatar
  • 326
3 votes
Accepted

Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram?

The reason most music-generation models use discrete representations is because the long-term structures of music are very challenging to model. Note that the MIDI data in MAESTRO (used in the two ...
Robz's user avatar
  • 204
2 votes
Accepted

Converting age and sex variables to a 64-unit dense layer

Convert them into numbers (using one-hot vectors or direct numerical representations) and then concatenate them. Then, you can pass them through the Dense layer.
Abhishek Verma's user avatar
2 votes

How does the embeddings work in vision transformer from paper?

In Machine Learning "embedding" means taking some set of raw inputs (like natural language tokens in NLP or image patches in your example) and converting them to vectors somehow. The ...
Kostya's user avatar
  • 2,496
2 votes
Accepted

What is the role of embeddings in a deep recurrent Q network?

The purpose of the input network is to embed the input tuple into a state/task representation, that can then be fed into the RNN hidden state at each time step. $(o^a_t,m^a′_{t−1},u^a_{t−1},a)$ (input)...
mugoh's user avatar
  • 531
2 votes

How to determine the embedding size?

I get an answer from this book: Machine Learning Design Patterns: Solutions to Common Challenges in Data Preparation, Model Building, and MLOps. If we’re in a hurry, one rule of thumb is to use the ...
Lerner Zhang's user avatar
2 votes

Is an embedding a representation of a word or its meaning?

Although we have had multiple similar questions (see here, here and here) and it seems to me that you focused on word embeddings (probably because you were not aware of the application of embeddings ...
nbro's user avatar
  • 40.1k
2 votes

Why are embeddings added, not concatenated?

First of all, I think it is very hard to properly reason about these things, but there are a few points that might justify using sum instead of concatenation. For example, concatenation would have the ...
Chillston's user avatar
  • 1,664
2 votes
Accepted

How embeddings learned from one model can be used in another?

An embedding layer is a linear layer that is used to convert a discrete input into a vector of a fixed size, d. Learned embedding layers are often used in natural language processing (NLP). Common ...
postnubilaphoebus's user avatar
2 votes
Accepted

How does GPT use the same embedding matrix for both input and output?

A GPT produces output based on its own previous output, so it must be able to understand its output. The learning input is provided as a stream of tokens, and these tokens are defined before learning ...
Volker Siegel's user avatar
2 votes
Accepted

What is a "continuous vector"?

Continuous vector simply means that each value of the vector is allowed to be a real number, in contrast of integer values used in other categorical encoding techniques like one-hot encoding (only 0 ...
Edoardo Guerriero's user avatar
2 votes

How embeddings learned from one model can be used in another?

To answer your one question: Are embeddings model-specific? YES! They are. I am not going to invoke math or other techniques here. My explanation is going to be in a intuitive perspective. I don't ...
dexterdev's user avatar
  • 121
2 votes

How do I choose a good treshold for classification (using cosine similarity scores)?

Determining the right classification/prediction threshold is always a trade-off between true positives, true negatives, false positives and false negatives. There is no universal guideline for ...
Robin van Hoorn's user avatar
2 votes

How do I choose a good treshold for classification (using cosine similarity scores)?

As correctly explained by @Robin van Hoorn, determining the classification threshold involves a trade-off between correct predictions and errors. One approach is to consider the TPR (true positive ...
Luca Anzalone's user avatar
2 votes

Match two paragraphs of text

There are several ways to do this. The most straightforward would be to encode the two paragraphs as vectors (also called text embeddings) using a pretrained language model. The idea is that the ...
Alexander Wan's user avatar
2 votes

From where do the Encoders in Transformers gets Input Embedding from?

Either by building embeddings yourself or loading pretrained embeddings. For building yourself, this is typically done with an auto-regressive model. It can be as simple as creating numeric ...
David Hoelzer's user avatar
1 vote

What is the distribution of autoencoder embeddings?

If you don't make assumptions about your input distribution, and the form of your network, it's very difficult to express the embedding distribution in closed-form. Here's an idea on how to formalize ...
Eduardo Montesuma's user avatar
1 vote

Why are embeddings added, not concatenated?

The confusion here is that we believe positional embedding is a more complicated version of adding positional information to the work embedding; however, it is not actually. Adding new dimensions to ...
Hamid Mohammadi's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible