Questions tagged [word-embedding]
For questions related to word embeddings, which are vector representations of words.
93
questions
0
votes
2
answers
204
views
From where do the Encoders in Transformers gets Input Embedding from?
In Transformers Encoders, from where do the Encoders get Input Embedding from?
So when a sentence is given to a transformer-based model it first tokenises the sentence and each token is mapped with ...
- 1
-1
votes
1
answer
239
views
How to get Llama-2 Rotary Embeddings?
I want to get the Llama-2 rotary embeddings. I do print(model) and get the following output:
In the picture I highlight the rotary embeddings.
How can get the ...
- 99
0
votes
1
answer
40
views
How word2vec de-embeds the special names in language models which output text
I am new to nlp field. I have some questions about word2vec embeddings. as I know they have a fixed size dictionary of vocabs. so definitely there some words which is not in that predefined dictionary ...
- 101
0
votes
0
answers
11
views
Update OpenAI embedding based on own domain corpus
I have a large domain corpus. Is there anyway to update the word/sentence/document embedding obtained from OpenAI embedding API based on my own domain corpus? There may be some words in my corpus that ...
- 101
0
votes
0
answers
26
views
Are the dimensions in embedding vectors ordered (similar to PCA)?
I am getting started with the vector embeddings. I have a general question about the embedding vectors generated by popular algorithms.
In PCA, usually, there is an implicit order of importance in the ...
- 101
0
votes
0
answers
9
views
Why skip gram doesnt just use the probabilities as the encoded vector?
I am very confused but this is what's on my head now:
The skip-gram algorithm just multiplies hot-encoded-words with a weights-matrix,
and since the word is hot encoded it is just multiplying a row ...
- 101
0
votes
0
answers
66
views
Word Embeddings but for Logical reasoning in custom knowledge GPT-3.5 bot
So I have created a chatbot using GPT-3.5 turbo. I have a vector database that holds vector embeddings of brands, ratings, commission percentages, outlets, tags, etc. Here's how the system is designed....
3
votes
1
answer
172
views
Would initializing transformers with pre-trained word embedding speed up the training of transformers?
I read the answers for that question What kind of word embedding is used in the original transformer?. It says that transforms like bert start the first word embedding layer with random values.
...
- 131
1
vote
1
answer
101
views
What is the intuition behind position-encoding?
It is clear that word positions are essential for the meaning of a sentence, and so are essential when feeding a sentence (= sequence of words) as a matrix of word embedding vectors into a transformer....
1
vote
1
answer
99
views
What information does the word embedding in Transformers will encode about the word when analysed outside of the model?
Word2vec and similar architectures create word embedding vectors as a byproduct from a supervised learning task, where they need to predict the correct context word. Consequently, the inner ...
- 35
0
votes
1
answer
27
views
Do different ngrams share embedding in Fasttext?
As per Section 3.2 in the original paper on Fasttext, the authors state:
In order to bound the memory requirements of our model, we use a
hashing function that maps n-grams to integers in 1 to K
...
0
votes
0
answers
26
views
How are the softmax normalized weights in ELMo actually learned and computed?
I was reading the ELMo paper, and they speak of task-specific representations of words (or tokens generally speaking) by using the following equation: $ELMo_{k}^{task} = \gamma^{task}\sum_{0}^{L}{s_{j}...
- 1
1
vote
0
answers
71
views
Is the input embedding split along the embedding dimension so that every head of the multi-head-attention module just gets a part of the input data?
So I found two contradictory explanations of the MHA (multi-head-self-attention-module):
In the first approach, the input embedding (= the input matrix) is split along the embedding dimension and all ...
- 11
0
votes
0
answers
43
views
References for the theory of pretraining and unsupervised learning to improve subsequent supervised learning
I am not sure if the title of this post uses the correct terminology, so suggestions are welcome.
I have been following a lot of the ideas of using Pre-training methods on neural networks, to improve ...
- 197
0
votes
0
answers
18
views
Does attention in transformers encode any information from positional embeddings?
I know we account for positional embeddings before feeding into attention layers, but would we be able to say that the Q and K dot products intrinsically encode relative positions
- 1
6
votes
2
answers
2k
views
"Attention is all you need" paper : How are the Q, K, V values calculated?
The seminal Attention is all you need paper introduces Transformers and implements the attention mecanism with "queries, keys, values", in an analogy to a retrieval system.
I understand the ...
- 221
0
votes
0
answers
61
views
How do you specify the dimension to search for similarity in CLIP image embeddings?
I have a question about CLIP semantic image search. When you have an image of a person e.g. a skinny person wearing red shirt, clip will search for you similarity in all dimensions including body ...
- 325
1
vote
1
answer
40
views
What's computationally more efficient between bag-of-words representation and bag-of-ngrams representation, with special regard to words order?
I cannot figure out what is more computationally efficient between the two representations mentioned in my question in terms of training time and the amount of data required. Especially, when it comes ...
0
votes
0
answers
40
views
I created joint embeddings by training a NN with contrastive loss. Why are my resulting embeddings so sparse?
Using BERT and Word2Vec word embeddings as two inputs, I trained a small neural network using Contrastive loss. The NN looks like this:
Net(
(fcin1): Linear(in_features=768, out_features=500, bias=...
0
votes
0
answers
240
views
How do transformers handle multidimensional input?
Transformers work with lists of vectors, i.e. sentence of length SEQ_LEN, with each word having size EMBEDDING_DIM. Now, since the model still makes use of Dense layers internally, i.e. as in https://...
1
vote
0
answers
95
views
Privacy implications of storing and transmitting GPT-3 embeddings?
We are exploring implementing a feature where a user might enter "which product has optional all wheel drive" into a search input, which would be transformed to GPT-3 embeddings, and ...
- 11
3
votes
2
answers
1k
views
How embeddings learned from one model can be used in another?
In the website the following explanation is provided about Embedding layer:
The Embedding layer is initialized with random weights and will learn
an embedding for all of the words in the training ...
- 43
0
votes
2
answers
723
views
How do Transformers compute the words embeddings at inference time since the embeddings are dynamic?
In Word2Vec, the embeddings don't depend on the context.
But in Transformers, the embeddings depend on the context.
So how are the words' embeddings set at inference time?
- 101
1
vote
2
answers
3k
views
How does GPT use the same embedding matrix for both input and output?
My understanding is that GPT uses the same embedding matrix for both inputs and output: Let $V$ be the vocab size, $D$ the number of embedding dimensions, and $E$ be a $V \times D$ embedding matrix:
...
- 63
0
votes
2
answers
151
views
What is the best way to create a vector representation (with fasttext) of a list of words?
Basically what I want to do is to create a single vector representation of a list of skills belonging to employees at a company (one list per employee). The embedding will be a representation of an ...
0
votes
1
answer
65
views
How is the training comlexity of NNLM word2vec calculated?
I was reading this paper on word2vec, and came around the following description of a feedforward NNLM:
It consists of input, projection, hidden and output layers. At the input layer, N previous words ...
1
vote
2
answers
79
views
Given embedding vector A and vector B, how to find top k embedding vectors such that they are similar to vector A and dissimilar to vector B
Which would be better approach for getting top k embedding vectors such that they are similar to embedding vector A and dissimilar to vector B.
Approach 1:
calculate ...
- 11
0
votes
0
answers
28
views
What are the types of inputs used for RNN in literature given sentences?
Suppose there are $m$ sentences in a text file and the number of distinct words is equal to $n$. The goal is to get word embeddings using RNN.
We know that it is impossible to pass any word, which is ...
- 3,612
1
vote
1
answer
1k
views
What exactly is embedding layer used in RNN encoders?
I am reading about RNN encoders. I came across the following line from this code. And I am facing difficulty in understanding the theoretical details regarding it.
...
- 3,612
1
vote
1
answer
357
views
General approaches in text encoding and labelling for NLP [closed]
What are the approaches of encoding text data? I would be glad to hear some summarization from experienced persons.
And are there any solutions accepting words outside the vocabulary and including ...
- 111
3
votes
1
answer
140
views
Why do we multipy context_size with embedding_dim? (PyTorch)
I've been using Tensorflow and just started learning PyTorch. I was following the tutorial: https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html#sphx-glr-beginner-nlp-word-...
- 145
4
votes
2
answers
4k
views
Can I always use "encoding" and "embedding" interchangeably?
This question is restricted to the text domain only.
The meaning of the word "encode" is Convert (information or instruction) into a particular form. One which performs encoding is called an ...
- 3,612
1
vote
0
answers
30
views
Why can't recurrent neural network handle large corpus for obtaining embeddings?
In order to learn the embeddings, we need to train a model based on some objective function. The model can be an RNN and the objective function can be the likelihood. We learn the embeddings by ...
- 3,612
1
vote
0
answers
99
views
How to generate text descriptions from keywords?
I wonder how can I build a neural network which will generate text description from given tag/tags. Let's assume I have created such data structure:
...
0
votes
2
answers
618
views
What is the exact difference between distributional semantics and distributed semantics?
While studying word embeddings in natural language processing, I encountered the following statement on page 327 of the textbook Natural Language Processing by Jacob Eisenstein
Distributional ...
- 3,612
1
vote
1
answer
211
views
Is categorical encoding a type of word embedding?
Word embedding refers to the techniques in which a word is represented by a vector. There are also integer encoding and one-hot encoding, which I will collectively call categorical encoding.
I can see ...
- 3,612
2
votes
1
answer
75
views
Why are BERT embeddings interpreted as representations of the corresponding words?
It's often assumed in literature that BERT embeddings are contextual representations of the corresponding word. That is, if the 5th word is "cold", then the 5th BERT embedding is a ...
1
vote
0
answers
51
views
How do sparse word embeddings fail to capture synonymy?
While reading some explanations of why dense word embeddings work better than sparse word embeddings, the following statement has been given in the chapter Vector Semantics and Embeddings, showing a ...
- 3,612
5
votes
2
answers
412
views
Is an embedding a representation of a word or its meaning?
What does the term "embedding" actually mean?
An embedding is a vector, but is that vector a representation of a word or its meaning? Literature loosely uses the word for both purposes. ...
- 3,612
0
votes
2
answers
340
views
Book(s) for text embedding
Text here refers to either character or word or sentence.
Is there any recent textbook that encompasses from classical methods to the modern techniques for embedding texts?
If a single textbook is ...
- 3,612
1
vote
2
answers
1k
views
Should I need to use BERT embeddings while tokenizing using BERT tokenizer?
I am new to BERT and NLP and I am a little confused with tokenization and word embedding.
My doubt is if I use the BertTokenizer for tokenizing a sentence then do I have to compulsorily use ...
- 111
5
votes
2
answers
7k
views
What is the difference between a language model and a word embedding?
I am self-studying applications of deep learning on the NLP and machine translation.
I am confused about the concepts of "Language Model", "Word Embedding", "BLEU Score".
...
- 293
1
vote
0
answers
44
views
Given the word embeddings, how do I create the sentence composed of the corresponding words?
I have done some reading. I want to implement an LSTM with pre-trained word embeddings (I also have plans to create my word embeddings, but let's cross that bridge when we come to it).
In any given ...
- 11
0
votes
1
answer
363
views
Are the Word2Vec encoded embeddings available online? [closed]
I am trying to do an NLP project and was wondering if there is anywhere online where the Word2Vec embeddings are stored (the actual n-dimmensional vectors).
I want to search up a word and see what its ...
- 15
0
votes
1
answer
109
views
NLP: Are hashtags tokenised?
I am exploring a potential NLP project. I was wondering what generally is done with the hashtags words (e.g. #hello). Are those words ignored? is the ...
- 15
1
vote
0
answers
513
views
What does the outputlayer of BERT for masked language modelling look like?
In the tutorial BERT – State of the Art Language Model for NLP the masked language modeling pre-training steps are described as follows:
In technical terms, the prediction of the output words ...
- 555
16
votes
3
answers
10k
views
What kind of word embedding is used in the original transformer?
I am currently trying to understand transformers.
To start, I read Attention Is All You Need and also this tutorial.
What makes me wonder is the word embedding used in the model. Is word2vec or GloVe ...
- 555
0
votes
0
answers
54
views
Is there a reason why no one combines word embeddings with the median?
Could you combine word embeddings with the median per dimension to get a document embedding? In my case I have a huge amount of words to build one document, which in turn should describe a topic. I ...
- 9
2
votes
0
answers
71
views
Can One-Hot Vectors be used as Inputs for Recurrent Neural Networks?
When using an RNN to encode a sentence, one normally takes each word, passes it through an embedding layer, and then uses the dense embedding as the input into the RNN.
Lets say instead of using dense ...
- 2,300
1
vote
2
answers
640
views
Why is an embedding of dimension 400 enough to represent 70000 words?
I am learning PyTorch on Udacity. In lesson 8, section 11: Training the Model, the instructor writes:
Then I have my embedding and hidden dimension. The embedding dimension is just a smaller ...
- 181