Questions tagged [word-embedding]

For questions related to word embeddings, which are vector representations of words.

Filter by
Sorted by
Tagged with
0 votes
0 answers
17 views

Self-Attention. Q, K, V and Word Embedding spaces

I've watched about 10 videos on the topic, and everyone presents their perspective on the subject. Various GPTs also can't seem to agree. So, I need to ask for a hard human knowledge. My question is: ...
Lex Podgorny's user avatar
0 votes
0 answers
23 views

Validating bundles of XMLs with inconsistent structure

The problem: We have a large number of XML bundles, and each bundle needs to meet certain criteria to be considered valid; specifically, certain values (or types of values) should belong to certain ...
SpaghettiM4ster's user avatar
1 vote
2 answers
93 views

How are sentences turned into a vector in LLM

My understanding of Large Language Models like GPT is that they are special kinds of deep neural networks specifically trained to predict the next word, given the beginning of a sentence. I understand ...
Weier's user avatar
  • 131
0 votes
1 answer
28 views

Why does skip-gram uses linear maps for embedding?

This question is inspired from Lilian Weng's blog here about skip-gram model, where she shows the model as multiplication via 2 matrices $W$ and $W'$ for embedding and word context matrix and $W$ ...
Mahammad Yusifov's user avatar
0 votes
0 answers
42 views

Concatenating the positional encoding

As per "Attention is all you need" etc., positional encoding is added to the embedded word vector input. My knee-jerk reaction is that this would muddle the "signal" of the word ...
SuaveSouris's user avatar
0 votes
2 answers
334 views

From where do the Encoders in Transformers gets Input Embedding from?

In Transformers Encoders, from where do the Encoders get Input Embedding from? So when a sentence is given to a transformer-based model it first tokenises the sentence and each token is mapped with ...
Swastik's user avatar
-1 votes
1 answer
575 views

How to get Llama-2 Rotary Embeddings?

I want to get the Llama-2 rotary embeddings. I do print(model) and get the following output: In the picture I highlight the rotary embeddings. How can get the ...
Christian01's user avatar
0 votes
1 answer
43 views

How word2vec de-embeds the special names in language models which output text

I am new to nlp field. I have some questions about word2vec embeddings. as I know they have a fixed size dictionary of vocabs. so definitely there some words which is not in that predefined dictionary ...
Farhang Amaji's user avatar
0 votes
0 answers
15 views

Update OpenAI embedding based on own domain corpus

I have a large domain corpus. Is there anyway to update the word/sentence/document embedding obtained from OpenAI embedding API based on my own domain corpus? There may be some words in my corpus that ...
Salty Gold Fish's user avatar
0 votes
0 answers
31 views

Are the dimensions in embedding vectors ordered (similar to PCA)?

I am getting started with the vector embeddings. I have a general question about the embedding vectors generated by popular algorithms. In PCA, usually, there is an implicit order of importance in the ...
JackDaniels's user avatar
0 votes
0 answers
10 views

Why skip gram doesnt just use the probabilities as the encoded vector?

I am very confused but this is what's on my head now: The skip-gram algorithm just multiplies hot-encoded-words with a weights-matrix, and since the word is hot encoded it is just multiplying a row ...
Minsky's user avatar
  • 101
0 votes
0 answers
73 views

Word Embeddings but for Logical reasoning in custom knowledge GPT-3.5 bot

So I have created a chatbot using GPT-3.5 turbo. I have a vector database that holds vector embeddings of brands, ratings, commission percentages, outlets, tags, etc. Here's how the system is designed....
Shahrukh Khan's user avatar
3 votes
1 answer
297 views

Would initializing transformers with pre-trained word embedding speed up the training of transformers?

I read the answers for that question What kind of word embedding is used in the original transformer?. It says that transforms like bert start the first word embedding layer with random values. ...
floyd's user avatar
  • 131
1 vote
1 answer
135 views

What is the intuition behind position-encoding?

It is clear that word positions are essential for the meaning of a sentence, and so are essential when feeding a sentence (= sequence of words) as a matrix of word embedding vectors into a transformer....
Hans-Peter Stricker's user avatar
1 vote
1 answer
134 views

What information does the word embedding in Transformers will encode about the word when analysed outside of the model?

Word2vec and similar architectures create word embedding vectors as a byproduct from a supervised learning task, where they need to predict the correct context word. Consequently, the inner ...
oliver.c's user avatar
  • 135
0 votes
1 answer
30 views

Do different ngrams share embedding in Fasttext?

As per Section 3.2 in the original paper on Fasttext, the authors state: In order to bound the memory requirements of our model, we use a hashing function that maps n-grams to integers in 1 to K ...
Fijoy Vadakkumpadan's user avatar
1 vote
0 answers
118 views

Is the input embedding split along the embedding dimension so that every head of the multi-head-attention module just gets a part of the input data?

So I found two contradictory explanations of the MHA (multi-head-self-attention-module): In the first approach, the input embedding (= the input matrix) is split along the embedding dimension and all ...
AAT's user avatar
  • 11
0 votes
0 answers
44 views

References for the theory of pretraining and unsupervised learning to improve subsequent supervised learning

I am not sure if the title of this post uses the correct terminology, so suggestions are welcome. I have been following a lot of the ideas of using Pre-training methods on neural networks, to improve ...
krishnab's user avatar
  • 207
8 votes
2 answers
5k views

"Attention is all you need" paper : How are the Q, K, V values calculated?

The seminal Attention is all you need paper (Google Brain team, 2017) introduces Transformers and implements the attention mecanism with "queries, keys, values", in an analogy to a retrieval ...
Soltius's user avatar
  • 271
1 vote
1 answer
47 views

What's computationally more efficient between bag-of-words representation and bag-of-ngrams representation, with special regard to words order?

I cannot figure out what is more computationally efficient between the two representations mentioned in my question in terms of training time and the amount of data required. Especially, when it comes ...
Filippo Colombi's user avatar
1 vote
0 answers
101 views

Privacy implications of storing and transmitting GPT-3 embeddings?

We are exploring implementing a feature where a user might enter "which product has optional all wheel drive" into a search input, which would be transformed to GPT-3 embeddings, and ...
jahghAAB's user avatar
3 votes
3 answers
2k views

How embeddings learned from one model can be used in another?

In the website the following explanation is provided about Embedding layer: The Embedding layer is initialized with random weights and will learn an embedding for all of the words in the training ...
Oculu 's user avatar
  • 43
0 votes
2 answers
1k views

How do Transformers compute the words embeddings at inference time since the embeddings are dynamic?

In Word2Vec, the embeddings don't depend on the context. But in Transformers, the embeddings depend on the context. So how are the words' embeddings set at inference time?
Julien's user avatar
  • 101
1 vote
2 answers
3k views

How does GPT use the same embedding matrix for both input and output?

My understanding is that GPT uses the same embedding matrix for both inputs and output: Let $V$ be the vocab size, $D$ the number of embedding dimensions, and $E$ be a $V \times D$ embedding matrix: ...
SRobertJames's user avatar
0 votes
2 answers
179 views

What is the best way to create a vector representation (with fasttext) of a list of words?

Basically what I want to do is to create a single vector representation of a list of skills belonging to employees at a company (one list per employee). The embedding will be a representation of an ...
Love Book's user avatar
0 votes
1 answer
70 views

How is the training comlexity of NNLM word2vec calculated?

I was reading this paper on word2vec, and came around the following description of a feedforward NNLM: It consists of input, projection, hidden and output layers. At the input layer, N previous words ...
MDescamps's user avatar
1 vote
2 answers
104 views

Given embedding vector A and vector B, how to find top k embedding vectors such that they are similar to vector A and dissimilar to vector B

Which would be better approach for getting top k embedding vectors such that they are similar to embedding vector A and dissimilar to vector B. Approach 1: calculate ...
Shubham's user avatar
  • 11
0 votes
0 answers
28 views

What are the types of inputs used for RNN in literature given sentences?

Suppose there are $m$ sentences in a text file and the number of distinct words is equal to $n$. The goal is to get word embeddings using RNN. We know that it is impossible to pass any word, which is ...
hanugm's user avatar
  • 3,840
1 vote
1 answer
2k views

What exactly is embedding layer used in RNN encoders?

I am reading about RNN encoders. I came across the following line from this code. And I am facing difficulty in understanding the theoretical details regarding it. ...
hanugm's user avatar
  • 3,840
1 vote
1 answer
409 views

General approaches in text encoding and labelling for NLP [closed]

What are the approaches of encoding text data? I would be glad to hear some summarization from experienced persons. And are there any solutions accepting words outside the vocabulary and including ...
Harry's user avatar
  • 111
3 votes
1 answer
149 views

Why do we multipy context_size with embedding_dim? (PyTorch)

I've been using Tensorflow and just started learning PyTorch. I was following the tutorial: https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html#sphx-glr-beginner-nlp-word-...
J.Smith's user avatar
  • 145
5 votes
3 answers
5k views

Can I always use "encoding" and "embedding" interchangeably?

This question is restricted to the text domain only. The meaning of the word "encode" is Convert (information or instruction) into a particular form. One which performs encoding is called an ...
hanugm's user avatar
  • 3,840
1 vote
0 answers
30 views

Why can't recurrent neural network handle large corpus for obtaining embeddings?

In order to learn the embeddings, we need to train a model based on some objective function. The model can be an RNN and the objective function can be the likelihood. We learn the embeddings by ...
hanugm's user avatar
  • 3,840
1 vote
0 answers
104 views

How to generate text descriptions from keywords?

I wonder how can I build a neural network which will generate text description from given tag/tags. Let's assume I have created such data structure: ...
Krzysztof Kaczyński's user avatar
0 votes
2 answers
689 views

What is the exact difference between distributional semantics and distributed semantics?

While studying word embeddings in natural language processing, I encountered the following statement on page 327 of the textbook Natural Language Processing by Jacob Eisenstein Distributional ...
hanugm's user avatar
  • 3,840
2 votes
1 answer
228 views

Is categorical encoding a type of word embedding?

Word embedding refers to the techniques in which a word is represented by a vector. There are also integer encoding and one-hot encoding, which I will collectively call categorical encoding. I can see ...
hanugm's user avatar
  • 3,840
2 votes
1 answer
89 views

Why are BERT embeddings interpreted as representations of the corresponding words?

It's often assumed in literature that BERT embeddings are contextual representations of the corresponding word. That is, if the 5th word is "cold", then the 5th BERT embedding is a ...
EmbeddingEnthusiast's user avatar
1 vote
0 answers
54 views

How do sparse word embeddings fail to capture synonymy?

While reading some explanations of why dense word embeddings work better than sparse word embeddings, the following statement has been given in the chapter Vector Semantics and Embeddings, showing a ...
hanugm's user avatar
  • 3,840
5 votes
2 answers
423 views

Is an embedding a representation of a word or its meaning?

What does the term "embedding" actually mean? An embedding is a vector, but is that vector a representation of a word or its meaning? Literature loosely uses the word for both purposes. ...
hanugm's user avatar
  • 3,840
0 votes
2 answers
413 views

Book(s) for text embedding

Text here refers to either character or word or sentence. Is there any recent textbook that encompasses from classical methods to the modern techniques for embedding texts? If a single textbook is ...
hanugm's user avatar
  • 3,840
1 vote
2 answers
1k views

Should I need to use BERT embeddings while tokenizing using BERT tokenizer?

I am new to BERT and NLP and I am a little confused with tokenization and word embedding. My doubt is if I use the BertTokenizer for tokenizing a sentence then do I have to compulsorily use ...
thenocturnalguy's user avatar
5 votes
2 answers
7k views

What is the difference between a language model and a word embedding?

I am self-studying applications of deep learning on the NLP and machine translation. I am confused about the concepts of "Language Model", "Word Embedding", "BLEU Score". ...
Exploring's user avatar
  • 343
1 vote
0 answers
46 views

Given the word embeddings, how do I create the sentence composed of the corresponding words?

I have done some reading. I want to implement an LSTM with pre-trained word embeddings (I also have plans to create my word embeddings, but let's cross that bridge when we come to it). In any given ...
trail99's user avatar
  • 11
0 votes
1 answer
385 views

Are the Word2Vec encoded embeddings available online? [closed]

I am trying to do an NLP project and was wondering if there is anywhere online where the Word2Vec embeddings are stored (the actual n-dimmensional vectors). I want to search up a word and see what its ...
user1234's user avatar
0 votes
1 answer
116 views

NLP: Are hashtags tokenised?

I am exploring a potential NLP project. I was wondering what generally is done with the hashtags words (e.g. #hello). Are those words ignored? is the ...
user1234's user avatar
1 vote
0 answers
579 views

What does the outputlayer of BERT for masked language modelling look like?

In the tutorial BERT – State of the Art Language Model for NLP the masked language modeling pre-training steps are described as follows: In technical terms, the prediction of the output words ...
Bert Gayus's user avatar
19 votes
3 answers
12k views

What kind of word embedding is used in the original transformer?

I am currently trying to understand transformers. To start, I read Attention Is All You Need and also this tutorial. What makes me wonder is the word embedding used in the model. Is word2vec or GloVe ...
Bert Gayus's user avatar
0 votes
0 answers
58 views

Is there a reason why no one combines word embeddings with the median?

Could you combine word embeddings with the median per dimension to get a document embedding? In my case I have a huge amount of words to build one document, which in turn should describe a topic. I ...
Stephan's user avatar
2 votes
0 answers
104 views

Can One-Hot Vectors be used as Inputs for Recurrent Neural Networks?

When using an RNN to encode a sentence, one normally takes each word, passes it through an embedding layer, and then uses the dense embedding as the input into the RNN. Lets say instead of using dense ...
chessprogrammer's user avatar
1 vote
2 answers
733 views

Why is an embedding of dimension 400 enough to represent 70000 words?

I am learning PyTorch on Udacity. In lesson 8, section 11: Training the Model, the instructor writes: Then I have my embedding and hidden dimension. The embedding dimension is just a smaller ...
Franva's user avatar
  • 181