37 votes
Accepted

What is the difference between latent and embedding spaces?

Embedding vs Latent Space Due to Machine Learning's recent and rapid renaissance, and the fact that it draws from many distinct areas of mathematics, statistics, and computer science, it often has a ...
brazofuerte's user avatar
  • 1,031
15 votes
Accepted

What kind of word embedding is used in the original transformer?

I have found a good answer in this blog post The Transformer: Attention Is All You Need: we learn a “word embedding” which is a smaller real-valued vector representation of the word that carries some ...
Bert Gayus's user avatar
6 votes
Accepted

What is the difference between a language model and a word embedding?

Simplified: Word Embeddings does not consider context, Language Models does. For e.g Word2Vec, GloVe, or fastText, there exists one fixed vector per word. Think of the following two sentences: The ...
Isbister's user avatar
  • 186
5 votes
Accepted

Is word embedding a form of feature extraction?

Though word-embedding is primarily a language modeling tool, it also acts as a feature extraction method because it helps transform raw data (characters in text documents) to a meaningful alignment of ...
Balraj Ashwath's user avatar
5 votes

What is the difference between latent and embedding spaces?

The expression "latent space" explicitly indicates that the space is associated with the mathematical concept of an hidden (or latent) variable, which cannot be observed directly, but only indirectly. ...
nbro's user avatar
  • 40.2k
5 votes

What is the intuition behind how word embeddings bring information to a neural network?

Shakespeare once said "A rose by any other name would smell as sweet" (Romeo and Juliet). Words are just labels we attach to ideas for convenience. By using one hot we remain tied to the letter ...
Colin Beckingham's user avatar
5 votes

What kind of word embedding is used in the original transformer?

No, neither Word2Vec nor GloVe is used as Transformers are a newer class of algorithms. Word2Vec and GloVe are based on static word embeddings while Transformers are based on dynamic word embeddings. ...
Brian O'Donnell's user avatar
5 votes
Accepted

Is an embedding a representation of a word or its meaning?

An embedding is a representation of a word that can be used as a proxy for some of its linguistic properties. The 'human' representation of a word, a sequence of letters and other symbols, is not ...
Oliver Mason's user avatar
  • 5,397
4 votes
Accepted

"Attention is all you need" paper : How are the Q, K, V values calculated?

(OP auto-answer) After having dug further in and read more papers on attention, and with help from Chillston in the comments, I think I've got it narrowed down to an issue of confusing notation. If ...
Soltius's user avatar
  • 241
3 votes

Adding BERT embeddings in LSTM embedding layer

Instead of using the Embedding() layer directly, you can create a new bertEmbedding() layer and use it instead. ...
skillsmuggler's user avatar
3 votes

Can ELMO embeddings be used to find the n most similar sentences?

I ended up finding this article which does what I'm looking for. Below is the portion of code I adapted for my needs ...
brokentransform's user avatar
3 votes

What is the intuition behind how word embeddings bring information to a neural network?

Adding to Colin's answer; using word embedding tend to be much more robust that one-hot vectors. Consider the the following two sentences: The desk has a book on it. and The table has a book on ...
Daniel's user avatar
  • 326
3 votes

What is the difference between a language model and a word embedding?

A language model aims to estimate the probability of one or more words given the surrounding words. Given a sentence composed of $w_{1},...,w_{i-1},\_ , w_{i+1},..,w_{n}$, you can find which is the i-...
SMattia's user avatar
  • 41
3 votes

Can I always use "encoding" and "embedding" interchangeably?

Caveat: I am not a native English speaker (but French). And mostly interested in symbolic artificial intelligence (the topic of my PhD thesis defended in 1990; see books by Jacques Pitrat) Encoding is ...
Basile Starynkevitch's user avatar
3 votes
Accepted

Why do we multipy context_size with embedding_dim? (PyTorch)

An n-gram language model is a language model trained with n context words. This means you're not feeding the model a single word but n. This is why the dimension of the input layer is "...
Edoardo Guerriero's user avatar
3 votes

What is the intuition behind position-encoding?

I understand the confusion. Although transformers are autoregressive (they predict something based on past information), they are not recurrent (do not have hidden states). In fact, think of a ...
Robin van Hoorn's user avatar
2 votes

Do individual dimensions in vector space have meaning?

Do individual dimensions in vector space have meaning? IIRC, some dimensions are interpretable, but in general this is not the case. Also it is debatable as to wether it is actually learning the ...
k.c. sayz 'k.c sayz''s user avatar
2 votes
Accepted

How is the word embedding represented in the paper "Recurrent neural network based language model"?

Input vector contains two concatenated parts. The low part represents the current word: word in time t encoded using 1-of-N coding [...] - size of vector x is equal to size of vocabulary V (this ...
pasaba por aqui's user avatar
2 votes

What is $USV^T$ in the context of word embeddings?

$USV^T$ refers to the result of the singular value decomposition (SVD). An $m \times n$ matrix $X$ can be written with the help of three matrices $$X = USV^T,$$ where $U$ is an $m \times m$ unitary ...
Ari Hietanen's user avatar
2 votes

Do we have cross-language vector space for word embedding?

You can try to read about MUSE (Multilingual Unsupervised and Supervised Embeddings) by Facebook. You can read it from its Github or this article. They also provide the FastText dictionary format (....
malioboro's user avatar
  • 2,819
2 votes

Does summing up word vectors destroy their meaning?

Summing up a sequence of word vector maybe used in practice sometimes. However, the operation of addition is non-reversible, meaning that once you sum up a few numbers, you cannot get the original ...
Clement's user avatar
  • 1,735
2 votes
Accepted

How does Continuous Bag of Words ensure that similar words are encoded as similar embeddings?

Unlike in skip-gram, the reason similar words have similar embeddings in CBOW is because the words show up in the same contexts of other skipped words. lets assume two words $e_i$ and $e_j$ pop up in ...
mshlis's user avatar
  • 2,359
2 votes

Is word embedding a form of feature extraction?

I think you guys are playing on semantics. If you consider feature extraction to be an unlearned preprocessing step to get inputs for your model, then no, word embeddings are not a feature extraction ...
mshlis's user avatar
  • 2,359
2 votes
Accepted

When to convert data to word embeddings in NLP

If you have to move a lot of data around during training (like retrieving batches from disk/network/what have you), it's much faster to do so as a rank-3 tensor of [batches, documents, indices] than ...
Andy's user avatar
  • 471
2 votes

Is there a pretrained (NLP) transformer that uses subword n-gram embeddings for tokenization like fasttext?

There is a pre-trained language model called ProphetNet for sequence-to-sequence learning with a novel self-supervised objective called future n-gram prediction. https://github.com/microsoft/...
usct01's user avatar
  • 121
2 votes

Is an embedding a representation of a word or its meaning?

Although we have had multiple similar questions (see here, here and here) and it seems to me that you focused on word embeddings (probably because you were not aware of the application of embeddings ...
nbro's user avatar
  • 40.2k
2 votes
Accepted

Is categorical encoding a type of word embedding?

One-hot encoding is different than the concept of a word embedding, although both approaches use vectors to represent the objects (e.g. words). A one-hot vector contains one element that is 1 and all ...
nbro's user avatar
  • 40.2k
2 votes

Can I always use "encoding" and "embedding" interchangeably?

From my experience with reading papers and books, I think these two terms are sometimes used interchangeably. As you also point out, an encoder (in an auto-encoder) also may also learn some "...
nbro's user avatar
  • 40.2k
2 votes
Accepted

How does GPT use the same embedding matrix for both input and output?

A GPT produces output based on its own previous output, so it must be able to understand its output. The learning input is provided as a stream of tokens, and these tokens are defined before learning ...
Volker Siegel's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible