35
votes
Accepted
What is the difference between latent and embedding spaces?
Embedding vs Latent Space
Due to Machine Learning's recent and rapid renaissance, and the fact that it draws from many distinct areas of mathematics, statistics, and computer science, it often has a ...
- 1,011
14
votes
Accepted
What kind of word embedding is used in the original transformer?
I have found a good answer in this blog post The Transformer: Attention Is All You Need:
we learn a “word embedding” which is a smaller real-valued vector representation of the word that carries some ...
- 555
5
votes
Accepted
Is word embedding a form of feature extraction?
Though word-embedding is primarily a language modeling tool, it also acts as a feature extraction method because it helps transform raw data (characters in text documents) to a meaningful alignment of ...
- 339
5
votes
What is the difference between latent and embedding spaces?
The expression "latent space" explicitly indicates that the space is associated with the mathematical concept of an hidden (or latent) variable, which cannot be observed directly, but only indirectly.
...
- 39.7k
5
votes
What is the intuition behind how word embeddings bring information to a neural network?
Shakespeare once said "A rose by any other name would smell as sweet" (Romeo and Juliet). Words are just labels we attach to ideas for convenience. By using one hot we remain tied to the letter ...
- 367
5
votes
What kind of word embedding is used in the original transformer?
No, neither Word2Vec nor GloVe is used as Transformers are a newer class of algorithms. Word2Vec and GloVe are based on static word embeddings while Transformers are based on dynamic word embeddings.
...
- 1,863
5
votes
Accepted
What is the difference between a language model and a word embedding?
Simplified: Word Embeddings does not consider context, Language Models does.
For e.g Word2Vec, GloVe, or fastText, there exists one fixed vector per word.
Think of the following two sentences:
The ...
- 176
5
votes
Accepted
Is an embedding a representation of a word or its meaning?
An embedding is a representation of a word that can be used as a proxy for some of its linguistic properties.
The 'human' representation of a word, a sequence of letters and other symbols, is not ...
- 5,382
3
votes
Adding BERT embeddings in LSTM embedding layer
Instead of using the Embedding() layer directly, you can create a new bertEmbedding() layer and use it instead.
...
- 393
3
votes
Can ELMO embeddings be used to find the n most similar sentences?
I ended up finding this article which does what I'm looking for.
Below is the portion of code I adapted for my needs
...
3
votes
What is the intuition behind how word embeddings bring information to a neural network?
Adding to Colin's answer; using word embedding tend to be much more robust that one-hot vectors. Consider the the following two sentences:
The desk has a book on it.
and
The table has a book on ...
- 326
3
votes
Can I always use "encoding" and "embedding" interchangeably?
Caveat: I am not a native English speaker (but French). And mostly interested in symbolic artificial intelligence (the topic of my PhD thesis defended in 1990; see books by Jacques Pitrat)
Encoding is ...
3
votes
Accepted
Why do we multipy context_size with embedding_dim? (PyTorch)
An n-gram language model is a language model trained with n context words. This means you're not feeding the model a single word but n. This is why the dimension of the input layer is "...
- 5,218
3
votes
Accepted
"Attention is all you need" paper : How are the Q, K, V values calculated?
(OP auto-answer) After having dug further in and read more papers on attention, and with help from Chillston in the comments, I think I've got it narrowed down to an issue of confusing notation. If ...
- 221
3
votes
What is the intuition behind position-encoding?
I understand the confusion. Although transformers are autoregressive (they predict something based on past information), they are not recurrent (do not have hidden states).
In fact, think of a ...
- 2,101
2
votes
Do individual dimensions in vector space have meaning?
Do individual dimensions in vector space have meaning?
IIRC, some dimensions are interpretable, but in general this is not the case. Also it is debatable as to wether it is actually learning the ...
- 2,071
2
votes
Accepted
How is the word embedding represented in the paper "Recurrent neural network based language model"?
Input vector contains two concatenated parts. The low part represents the current word:
word in time t encoded using 1-of-N coding [...] - size of vector x is
equal to size of vocabulary V (this ...
- 1,283
2
votes
What is $USV^T$ in the context of word embeddings?
$USV^T$ refers to the result of the singular value decomposition (SVD).
An $m \times n$ matrix $X$ can be written with the help of three matrices
$$X = USV^T,$$
where $U$ is an $m \times m$ unitary ...
- 121
2
votes
Do we have cross-language vector space for word embedding?
You can try to read about MUSE (Multilingual Unsupervised and Supervised Embeddings) by Facebook. You can read it from its Github or this article. They also provide the FastText dictionary format (....
- 2,789
2
votes
Does summing up word vectors destroy their meaning?
Summing up a sequence of word vector maybe used in practice sometimes. However, the operation of addition is non-reversible, meaning that once you sum up a few numbers, you cannot get the original ...
- 1,735
2
votes
Accepted
How does Continuous Bag of Words ensure that similar words are encoded as similar embeddings?
Unlike in skip-gram, the reason similar words have similar embeddings in CBOW is because the words show up in the same contexts of other skipped words.
lets assume two words $e_i$ and $e_j$ pop up in ...
- 2,359
2
votes
Is word embedding a form of feature extraction?
I think you guys are playing on semantics.
If you consider feature extraction to be an unlearned preprocessing step to get inputs for your model, then no, word embeddings are not a feature extraction ...
- 2,359
2
votes
Accepted
When to convert data to word embeddings in NLP
If you have to move a lot of data around during training (like retrieving batches from disk/network/what have you), it's much faster to do so as a rank-3 tensor of [batches, documents, indices] than ...
- 436
2
votes
Is there a pretrained (NLP) transformer that uses subword n-gram embeddings for tokenization like fasttext?
There is a pre-trained language model called ProphetNet for sequence-to-sequence learning with a novel self-supervised objective called future n-gram prediction.
https://github.com/microsoft/...
- 121
2
votes
What is the difference between a language model and a word embedding?
A language model aims to estimate the probability of one or more words given the surrounding words. Given a sentence composed of $w_{1},...,w_{i-1},\_ , w_{i+1},..,w_{n}$, you can find which is the i-...
- 21
2
votes
Is an embedding a representation of a word or its meaning?
Although we have had multiple similar questions (see here, here and here) and it seems to me that you focused on word embeddings (probably because you were not aware of the application of embeddings ...
- 39.7k
2
votes
Accepted
Is categorical encoding a type of word embedding?
One-hot encoding is different than the concept of a word embedding, although both approaches use vectors to represent the objects (e.g. words).
A one-hot vector contains one element that is 1 and all ...
- 39.7k
2
votes
Can I always use "encoding" and "embedding" interchangeably?
From my experience with reading papers and books, I think these two terms are sometimes used interchangeably.
As you also point out, an encoder (in an auto-encoder) also may also learn some "...
- 39.7k
2
votes
Accepted
How does GPT use the same embedding matrix for both input and output?
A GPT produces output based on its own previous output, so it must be able to understand its output.
The learning input is provided as a stream of tokens, and these tokens are defined before learning ...
- 667
Only top scored, non community-wiki answers of a minimum length are eligible