26 votes
Accepted

What is the difference between latent and embedding spaces?

Embedding vs Latent Space Due to Machine Learning's recent and rapid renaissance, and the fact that it draws from many distinct areas of mathematics, statistics, and computer science, it often has a ...
8 votes
Accepted

What kind of word embedding is used in the original transformer?

I have found a good answer in this blog post The Transformer: Attention Is All You Need: we learn a “word embedding” which is a smaller real-valued vector representation of the word that carries some ...
5 votes
Accepted

Is word embedding a form of feature extraction?

Though word-embedding is primarily a language modeling tool, it also acts as a feature extraction method because it helps transform raw data (characters in text documents) to a meaningful alignment of ...
5 votes

What is the intuition behind how word embeddings bring information to a neural network?

Shakespeare once said "A rose by any other name would smell as sweet" (Romeo and Juliet). Words are just labels we attach to ideas for convenience. By using one hot we remain tied to the letter ...
5 votes
Accepted

Is an embedding a representation of a word or its meaning?

An embedding is a representation of a word that can be used as a proxy for some of its linguistic properties. The 'human' representation of a word, a sequence of letters and other symbols, is not ...
  • 5,187
4 votes

What is the difference between latent and embedding spaces?

The expression "latent space" explicitly indicates that the space is associated with the mathematical concept of an hidden (or latent) variable, which cannot be observed directly, but only indirectly. ...
  • 35.6k
4 votes

What kind of word embedding is used in the original transformer?

No, neither Word2Vec nor GloVe is used as Transformers are a newer class of algorithms. Word2Vec and GloVe are based on static word embeddings while Transformers are based on dynamic word embeddings. ...
4 votes
Accepted

What is the difference between a language model and a word embedding?

Simplified: Word Embeddings does not consider context, Language Models does. For e.g Word2Vec, GloVe, or fastText, there exists one fixed vector per word. Think of the following two sentences: The ...
  • 156
3 votes

Adding BERT embeddings in LSTM embedding layer

Instead of using the Embedding() layer directly, you can create a new bertEmbedding() layer and use it instead. ...
3 votes

What is the intuition behind how word embeddings bring information to a neural network?

Adding to Colin's answer; using word embedding tend to be much more robust that one-hot vectors. Consider the the following two sentences: The desk has a book on it. and The table has a book on ...
  • 326
3 votes
Accepted

Why do we multipy context_size with embedding_dim? (PyTorch)

An n-gram language model is a language model trained with n context words. This means you're not feeding the model a single word but n. This is why the dimension of the input layer is "...
2 votes

Do individual dimensions in vector space have meaning?

Do individual dimensions in vector space have meaning? IIRC, some dimensions are interpretable, but in general this is not the case. Also it is debatable as to wether it is actually learning the ...
2 votes
Accepted

How is the word embedding represented in the paper "Recurrent neural network based language model"?

Input vector contains two concatenated parts. The low part represents the current word: word in time t encoded using 1-of-N coding [...] - size of vector x is equal to size of vocabulary V (this ...
2 votes

What is $USV^T$ in the context of word embeddings?

$USV^T$ refers to the result of the singular value decomposition (SVD). An $m \times n$ matrix $X$ can be written with the help of three matrices $$X = USV^T,$$ where $U$ is an $m \times m$ unitary ...
2 votes

Do we have cross-language vector space for word embedding?

You can try to read about MUSE (Multilingual Unsupervised and Supervised Embeddings) by Facebook. You can read it from its Github or this article. They also provide the FastText dictionary format (....
  • 2,589
2 votes
Accepted

How does Continuous Bag of Words ensure that similar words are encoded as similar embeddings?

Unlike in skip-gram, the reason similar words have similar embeddings in CBOW is because the words show up in the same contexts of other skipped words. lets assume two words $e_i$ and $e_j$ pop up in ...
  • 2,329
2 votes

Can ELMO embeddings be used to find the n most similar sentences?

I ended up finding this article which does what I'm looking for. Below is the portion of code I adapted for my needs ...
2 votes

Does summing up word vectors destroy their meaning?

Summing up a sequence of word vector maybe used in practice sometimes. However, the operation of addition is non-reversible, meaning that once you sum up a few numbers, you cannot get the original ...
  • 1,715
2 votes

Is word embedding a form of feature extraction?

I think you guys are playing on semantics. If you consider feature extraction to be an unlearned preprocessing step to get inputs for your model, then no, word embeddings are not a feature extraction ...
  • 2,329
2 votes
Accepted

When to convert data to word embeddings in NLP

If you have to move a lot of data around during training (like retrieving batches from disk/network/what have you), it's much faster to do so as a rank-3 tensor of [batches, documents, indices] than ...
  • 426
2 votes

Is there a pretrained (NLP) transformer that uses subword n-gram embeddings for tokenization like fasttext?

There is a pre-trained language model called ProphetNet for sequence-to-sequence learning with a novel self-supervised objective called future n-gram prediction. https://github.com/microsoft/...
  • 121
2 votes

What is the difference between a language model and a word embedding?

A language model aims to estimate the probability of one or more words given the surrounding words. Given a sentence composed of $w_{1},...,w_{i-1},\_ , w_{i+1},..,w_{n}$, you can find which is the i-...
  • 21
2 votes

Is an embedding a representation of a word or its meaning?

Although we have had multiple similar questions (see here, here and here) and it seems to me that you focused on word embeddings (probably because you were not aware of the application of embeddings ...
  • 35.6k
2 votes
Accepted

Is categorical encoding a type of word embedding?

One-hot encoding is different than the concept of a word embedding, although both approaches use vectors to represent the objects (e.g. words). A one-hot vector contains one element that is 1 and all ...
  • 35.6k
1 vote

When to convert data to word embeddings in NLP

There are multiple ways to get word embedding from a corpus. Count Vectorizer: You can use the CountVectorizer() from ...
1 vote
Accepted

How is dropout applied to the embedding layer's output?

It doesn't drops rows or columns, it acts directly on scalars. The Dropout Layer keras documentation explains it and illustrates it with an example : The Dropout layer randomly sets input units to 0 ...
  • 571
1 vote

How can I create an embedding layer to convert words to a vector space from scratch?

Word2vec embeddings are trained using a simple auto-encoder model that takes a word and tries to predict one word from the window of surrounding words. You could define it like this: ...
1 vote

How to add a pretrained model to my layers to get embeddings?

I think you should use Keras embedding layer. It will be too easier than what you are doing. Steps Create Embedding Matrix add matrix to embedding layer while building model. You will find detailed ...
  • 111
1 vote

What should the dimension of the input be for text summarization?

Briefly: I think what you are looking for is an RNN network (Either LSTM or GRU) with the many-to-many topology. Explanation: Clearly your input is the sentences (or to be more precise, the an ...
  • 395

Only top scored, non community-wiki answers of a minimum length are eligible