Questions tagged [word-embedding]

For questions related to word embeddings, which are vector representations of words.

Filter by
Sorted by
Tagged with
1 vote
1 answer
16 views

What's computationally more efficient between bag-of-words representation and bag-of-ngrams representation, with special regard to words order?

I cannot figure out what is more computationally efficient between the two representations mentioned in my question in terms of training time and the amount of data required. Especially, when it comes ...
0 votes
0 answers
6 views

I created joint embeddings by training a NN with contrastive loss. Why are my resulting embeddings so sparse?

Using BERT and Word2Vec word embeddings as two inputs, I trained a small neural network using Contrastive loss. The NN looks like this: Net( (fcin1): Linear(in_features=768, out_features=500, bias=...
0 votes
0 answers
23 views

How do transformers handle multidimensional input?

Transformers work with lists of vectors, i.e. sentence of length SEQ_LEN, with each word having size EMBEDDING_DIM. Now, since the model still makes use of Dense layers internally, i.e. as in https://...
1 vote
0 answers
30 views

Privacy implications of storing and transmitting GPT-3 embeddings?

We are exploring implementing a feature where a user might enter "which product has optional all wheel drive" into a search input, which would be transformed to GPT-3 embeddings, and ...
1 vote
1 answer
108 views

How embeddings learned from one model can be used in another?

In the website the following explanation is provided about Embedding layer: The Embedding layer is initialized with random weights and will learn an embedding for all of the words in the training ...
  • 23
0 votes
2 answers
127 views

How do Transformers compute the words embeddings at inference time since the embeddings are dynamic?

In Word2Vec, the embeddings don't depend on the context. But in Transformers, the embeddings depend on the context. So how are the words' embeddings set at inference time?
  • 101
0 votes
0 answers
19 views

How to Create a Fixed-Length, Binary, Sequence of Tokens Embedding?

Say I have ten classes represented by 1 x n_classes vector of binary. My goal is to embed a sequence of 1xN binary so that I can also model the class-co occurrence. Say, class A, B, D are present and ...
0 votes
0 answers
97 views

Do Transformers and LSTMs use the same word embeddings (except for the position encoding, which only Transformers use)?

In NLP, the first step is always to "convert" the given words of a sentence into representation vectors (word embeddings). As I understand it, in the case of transformers, the words/...
  • 1
0 votes
0 answers
56 views

Is BERT capable of producing semantically close word embeddings for synonyms?

I am currently working on my undergraduate thesis on matching job descriptions to resumes based on the contents of both. Recently, I came across the following statement by Schmitt et al., 2016: "[...
0 votes
1 answer
844 views

How does GPT use the same embedding matrix for both input and output?

My understanding is that GPT uses the same embedding matrix for both inputs and output: Let $V$ be the vocab size, $D$ the number of embedding dimensions, and $E$ be a $V \times D$ embedding matrix: ...
0 votes
2 answers
66 views

What is the best way to create a vector representation (with fasttext) of a list of words?

Basically what I want to do is to create a single vector representation of a list of skills belonging to employees at a company (one list per employee). The embedding will be a representation of an ...
0 votes
1 answer
41 views

How is the training comlexity of NNLM word2vec calculated?

I was reading this paper on word2vec, and came around the following description of a feedforward NNLM: It consists of input, projection, hidden and output layers. At the input layer, N previous words ...
0 votes
0 answers
77 views

Neural Network with numerical data and sentences as features

I'm beginning in the words A.I. features. My current problem is that I want to create Neural Network that takes as input numerical data and also words as data (by words, I mean multiple sentences) to ...
1 vote
2 answers
52 views

Given embedding vector A and vector B, how to find top k embedding vectors such that they are similar to vector A and dissimilar to vector B

Which would be better approach for getting top k embedding vectors such that they are similar to embedding vector A and dissimilar to vector B. Approach 1: calculate ...
  • 11
0 votes
0 answers
27 views

What are the types of inputs used for RNN in literature given sentences?

Suppose there are $m$ sentences in a text file and the number of distinct words is equal to $n$. The goal is to get word embeddings using RNN. We know that it is impossible to pass any word, which is ...
  • 3,451
1 vote
1 answer
751 views

What exactly is embedding layer used in RNN encoders?

I am reading about RNN encoders. I came across the following line from this code. And I am facing difficulty in understanding the theoretical details regarding it. ...
  • 3,451
1 vote
1 answer
187 views

General approaches in text encoding and labelling for NLP [closed]

What are the approaches of encoding text data? I would be glad to hear some summarization from experienced persons. And are there any solutions accepting words outside the vocabulary and including ...
  • 111
3 votes
1 answer
119 views

Why do we multipy context_size with embedding_dim? (PyTorch)

I've been using Tensorflow and just started learning PyTorch. I was following the tutorial: https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html#sphx-glr-beginner-nlp-word-...
  • 145
2 votes
2 answers
1k views

Can I always use "encoding" and "embedding" interchangeably?

This question is restricted to the text domain only. The meaning of the word "encode" is Convert (information or instruction) into a particular form. One which performs encoding is called an ...
  • 3,451
1 vote
0 answers
28 views

Why can't recurrent neural network handle large corpus for obtaining embeddings?

In order to learn the embeddings, we need to train a model based on some objective function. The model can be an RNN and the objective function can be the likelihood. We learn the embeddings by ...
  • 3,451
1 vote
0 answers
74 views

How to generate text descriptions from keywords?

I wonder how can I build a neural network which will generate text description from given tag/tags. Let's assume I have created such data structure: ...
0 votes
2 answers
391 views

What is the exact difference between distributional semantics and distributed semantics?

While studying word embeddings in natural language processing, I encountered the following statement on page 327 of the textbook Natural Language Processing by Jacob Eisenstein Distributional ...
  • 3,451
1 vote
1 answer
147 views

Is categorical encoding a type of word embedding?

Word embedding refers to the techniques in which a word is represented by a vector. There are also integer encoding and one-hot encoding, which I will collectively call categorical encoding. I can see ...
  • 3,451
1 vote
0 answers
39 views

Why are BERT embeddings interpreted as representations of the corresponding words?

It's often assumed in literature that BERT embeddings are contextual representations of the corresponding word. That is, if the 5th word is "cold", then the 5th BERT embedding is a ...
1 vote
0 answers
45 views

How do sparse word embeddings fail to capture synonymy?

While reading some explanations of why dense word embeddings work better than sparse word embeddings, the following statement has been given in the chapter Vector Semantics and Embeddings, showing a ...
  • 3,451
5 votes
2 answers
372 views

Is an embedding a representation of a word or its meaning?

What does the term "embedding" actually mean? An embedding is a vector, but is that vector a representation of a word or its meaning? Literature loosely uses the word for both purposes. ...
  • 3,451
0 votes
2 answers
140 views

Book(s) for text embedding

Text here refers to either character or word or sentence. Is there any recent textbook that encompasses from classical methods to the modern techniques for embedding texts? If a single textbook is ...
  • 3,451
1 vote
2 answers
746 views

Should I need to use BERT embeddings while tokenizing using BERT tokenizer?

I am new to BERT and NLP and I am a little confused with tokenization and word embedding. My doubt is if I use the BertTokenizer for tokenizing a sentence then do I have to compulsorily use ...
4 votes
2 answers
4k views

What is the difference between a language model and a word embedding?

I am self-studying applications of deep learning on the NLP and machine translation. I am confused about the concepts of "Language Model", "Word Embedding", "BLEU Score". ...
  • 133
1 vote
0 answers
28 views

Given the word embeddings, how do I create the sentence composed of the corresponding words?

I have done some reading. I want to implement an LSTM with pre-trained word embeddings (I also have plans to create my word embeddings, but let's cross that bridge when we come to it). In any given ...
  • 11
0 votes
1 answer
226 views

Are the Word2Vec encoded embeddings available online? [closed]

I am trying to do an NLP project and was wondering if there is anywhere online where the Word2Vec embeddings are stored (the actual n-dimmensional vectors). I want to search up a word and see what its ...
0 votes
1 answer
80 views

NLP: Are hashtags tokenised?

I am exploring a potential NLP project. I was wondering what generally is done with the hashtags words (e.g. #hello). Are those words ignored? is the ...
1 vote
0 answers
318 views

What does the outputlayer of BERT for masked language modelling look like?

In the tutorial BERT – State of the Art Language Model for NLP the masked language modeling pre-training steps are described as follows: In technical terms, the prediction of the output words ...
11 votes
3 answers
5k views

What kind of word embedding is used in the original transformer?

I am currently trying to understand transformers. To start, I read Attention Is All You Need and also this tutorial. What makes me wonder is the word embedding used in the model. Is word2vec or GloVe ...
0 votes
0 answers
46 views

Is there a reason why no one combines word embeddings with the median?

Could you combine word embeddings with the median per dimension to get a document embedding? In my case I have a huge amount of words to build one document, which in turn should describe a topic. I ...
2 votes
0 answers
45 views

Can One-Hot Vectors be used as Inputs for Recurrent Neural Networks?

When using an RNN to encode a sentence, one normally takes each word, passes it through an embedding layer, and then uses the dense embedding as the input into the RNN. Lets say instead of using dense ...
1 vote
2 answers
338 views

Why is an embedding of dimension 400 enough to represent 70000 words?

I am learning PyTorch on Udacity. In lesson 8, section 11: Training the Model, the instructor writes: Then I have my embedding and hidden dimension. The embedding dimension is just a smaller ...
  • 181
4 votes
1 answer
328 views

Is there a pretrained (NLP) transformer that uses subword n-gram embeddings for tokenization like fasttext?

I know that several tokenization methods that are used for tranformer models like WordPiece for Bert and BPE for Roberta and others. What I was wondering if there is also a transformer which uses a ...
  • 41
1 vote
0 answers
20 views

Bechmark models for Text Classification / Sentiment Classification

I am currently working on a novel application in NLP where I try to classify empathic and non-empathic texts. I would like to compare the performance of my model to some benchmark models. As I am ...
  • 11
1 vote
0 answers
200 views

How homographs is an NLP task can be treated?

A homograph - is a word that shares the same written form as another word but has a different meaning. They can be even different parts of speech. For example: close(verb) - close(adverb) lead(verb)...
3 votes
3 answers
123 views

When to convert data to word embeddings in NLP

When training a network using word embeddings, it is standard to add an embedding layer to first convert the input vector to the embeddings. However, assuming the embeddings are pre-trained and frozen,...
1 vote
1 answer
2k views

How is dropout applied to the embedding layer's output?

...
  • 197
1 vote
0 answers
88 views

Is it good practice to save NLP Transformer based pre-trained models into file system in production environment

I have developed a multi label classifier using BERT. I'm leveraging Hugging Face Pytorch implementation for transformers. I have saved the pretrained model into the file directory in dev environment. ...
  • 141
3 votes
1 answer
80 views

How can I create an embedding layer to convert words to a vector space from scratch?

For an upcoming project, I am trying to build a neural network for classifying text from scratch, without the use of libraries. This requires an embedding layer, or a way to convert words to some ...
0 votes
1 answer
157 views

How to add a pretrained model to my layers to get embeddings?

I want to use a pretrained model found in [BERT Embeddings] https://github.com/UKPLab/sentence-transformers and I want to add a layer to get the sentence embeddings from the model and pass on to the ...
1 vote
0 answers
43 views

How many spectrogram frames per input character does text-to-speech (TTS) system Tacotron-2 generate?

I've been reading on Tacotron-2, a text-to-speech system, that generates speech just-like humans (indistinguishable from humans) using the GitHub https://github.com/Rayhane-mamah/Tacotron-2. I'm very ...
  • 171
3 votes
1 answer
55 views

What should the dimension of the input be for text summarization?

I am trying to build a model for extractive text summarization using keras sequential layers. I am having a hard time trying to understand how to input my x data. Should it be an array of documents ...
1 vote
0 answers
55 views

Creating Text Features using word2vec

My task is to classify some texts. I have used word2vec to represent text words and I pass them to an LSTM as input. Taking into account that texts do not contain the same number of words, is it a ...
  • 909
3 votes
0 answers
97 views

Is there a good book or paper on word embeddings?

Is there a good and modern book that focuses on word embeddings and their applications? It would also be ok to provide the name of a paper that provides a good overview of word embeddings.
  • 909
1 vote
1 answer
42 views

Why I have a different number of terms in word2vec and TFIDF? How I can fix it?

I need multiply the weigths of terms in TFIDF matrix by the word-embeddings of word2vec matrix but I can't do it because each matrix have a different number of terms. I am using the same corpus for ...
  • 13