37
votes
Accepted
What is the difference between latent and embedding spaces?
Embedding vs Latent Space
Due to Machine Learning's recent and rapid renaissance, and the fact that it draws from many distinct areas of mathematics, statistics, and computer science, it often has a ...
- 1,031
7
votes
Why are embeddings added, not concatenated?
In high-dimensional spaces, the token embeddings and positional encodings can be thought of as forming two separate subspaces that are approximately orthogonal to each other. This is based on the ...
- 169
7
votes
How to determine the embedding size?
There is a rule of thumb that says min(50, num_categories/2). But this tops out at 100 categories, what to do after that? I propose this:
When num_categories <= 1000:
...
7
votes
How to determine the embedding size?
In most cases, seems that embedding dim is chosen empirically, by trial and error.
Older papers in NLP used 300 conventionally https://petuum.medium.com/embeddings-a-matrix-of-meaning-4de877c9aa27. ...
- 2,754
5
votes
What is the difference between latent and embedding spaces?
The expression "latent space" explicitly indicates that the space is associated with the mathematical concept of an hidden (or latent) variable, which cannot be observed directly, but only indirectly.
...
- 41.4k
5
votes
What is the intuition behind how word embeddings bring information to a neural network?
Shakespeare once said "A rose by any other name would smell as sweet" (Romeo and Juliet). Words are just labels we attach to ideas for convenience. By using one hot we remain tied to the letter ...
- 367
5
votes
Accepted
Is an embedding a representation of a word or its meaning?
An embedding is a representation of a word that can be used as a proxy for some of its linguistic properties.
The 'human' representation of a word, a sequence of letters and other symbols, is not ...
- 5,417
5
votes
Creating a support chat bot for my business
First and foremost, do not use GPT/OpenAI for customer-facing applications. You end up with a mess. GPT is great for creative work, but not for production. GPT is a probabilistic language model, and ...
- 53
3
votes
Accepted
What is the difference between representation and embedding?
Vector representation is a generic term used to talk about any type of feature encoding, embedding vectors are instead a special case of vector representation.
When talking about vector representation ...
- 5,428
3
votes
Sequence Embedding using embedding layer: how does the network architecture influence it?
Premises: mine is not gonna be an exhaustive answer, also I'm more familiar with classic natural language processing than with embedding vectors applied to protein sequences. Said so, I think I can ...
- 5,428
3
votes
Accepted
What are knowledge graph embeddings?
Knowledge graph embeddings (KGE) are embeddings created in the context of a knowledge graph (KG), which can be viewed as a visual/graphical representation of a knowledge base, where nodes are entities ...
- 41.4k
3
votes
How to determine the embedding size?
I get an answer from this book: Machine Learning Design Patterns: Solutions to Common Challenges in Data Preparation, Model Building, and MLOps.
If we’re in a hurry, one rule of thumb is to use the ...
- 967
3
votes
Accepted
Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram?
The reason most music-generation models use discrete representations is because the long-term structures of music are very challenging to model. Note that the MIDI data in MAESTRO (used in the two ...
- 204
3
votes
What is the intuition behind how word embeddings bring information to a neural network?
Adding to Colin's answer; using word embedding tend to be much more robust that one-hot vectors. Consider the the following two sentences:
The desk has a book on it.
and
The table has a book on ...
- 326
3
votes
How embeddings learned from one model can be used in another?
To answer your one question: Are embeddings model-specific? YES! They are. I am not going to invoke math or other techniques here. My explanation is going to be in a intuitive perspective. I don't ...
- 131
2
votes
Accepted
What is the role of embeddings in a deep recurrent Q network?
The purpose of the input network is to embed the input tuple into a state/task representation, that can then be fed into the RNN hidden state at each time step.
$(o^a_t,m^a′_{t−1},u^a_{t−1},a)$ (input)...
- 541
2
votes
Accepted
Converting age and sex variables to a 64-unit dense layer
Convert them into numbers (using one-hot vectors or direct numerical representations) and then concatenate them. Then, you can pass them through the Dense layer.
- 868
2
votes
How does the embeddings work in vision transformer from paper?
In Machine Learning "embedding" means taking some set of raw inputs (like natural language tokens in NLP or image patches in your example) and converting them to vectors somehow. The ...
- 2,562
2
votes
Is an embedding a representation of a word or its meaning?
Although we have had multiple similar questions (see here, here and here) and it seems to me that you focused on word embeddings (probably because you were not aware of the application of embeddings ...
- 41.4k
2
votes
Why are embeddings added, not concatenated?
First of all, I think it is very hard to properly reason about these things, but there are a few points that might justify using sum instead of concatenation.
For example, concatenation would have the ...
- 1,748
2
votes
Accepted
How does GPT use the same embedding matrix for both input and output?
A GPT produces output based on its own previous output, so it must be able to understand its output.
The learning input is provided as a stream of tokens, and these tokens are defined before learning ...
- 767
2
votes
Accepted
What is a "continuous vector"?
Continuous vector simply means that each value of the vector is allowed to be a real number, in contrast of integer values used in other categorical encoding techniques like one-hot encoding (only 0 ...
- 5,428
2
votes
Accepted
How embeddings learned from one model can be used in another?
An embedding layer is a linear layer that is used to convert a discrete input into a vector of a fixed size, d. Learned embedding layers are often used in natural language processing (NLP). Common ...
2
votes
How can I send vectors as a chat context?
Do you mean sending the vector of embedded text as the context, instead of the text itself?
If you think it through that might mean sending in ...
- 529
2
votes
How do I choose a good treshold for classification (using cosine similarity scores)?
Determining the right classification/prediction threshold is always a trade-off between true positives, true negatives, false positives and false negatives. There is no universal guideline for ...
- 2,660
2
votes
How do I choose a good treshold for classification (using cosine similarity scores)?
As correctly explained by @Robin van Hoorn, determining the classification threshold involves a trade-off between correct predictions and errors.
One approach is to consider the TPR (true positive ...
- 3,186
2
votes
Match two paragraphs of text
There are several ways to do this. The most straightforward would be to encode the two paragraphs as vectors (also called text embeddings) using a pretrained language model.
The idea is that the ...
- 1,399
2
votes
How can I teach a book to an LLM?
Using embeddings is an effective approach when you have limited data, such as a book, and want to extract relevant context and related text for querying.
An example of this approach at this URL: https:...
- 21
2
votes
From where do the Encoders in Transformers gets Input Embedding from?
Either by building embeddings yourself or loading pretrained embeddings.
For building yourself, this is typically done with an auto-regressive model. It can be as simple as creating numeric ...
- 953
Only top scored, non community-wiki answers of a minimum length are eligible