Questions tagged [word-embedding]

For questions related to word embeddings, which are vector representations of words.

Filter by
Sorted by
Tagged with
0 votes
2 answers
34 views

What is the best way to create a vector representation (with fasttext) of a list of words?

Basically what I want to do is to create a single vector representation of a list of skills belonging to employees at a company (one list per employee). The embedding will be a representation of an ...
user avatar
0 votes
0 answers
15 views

How is the training comlexity of NNLM word2vec calculated?

I was reading this paper on word2vec, and came around the following description of a feedforward NNLM: It consists of input, projection, hidden and output layers. At the input layer, N previous words ...
user avatar
0 votes
0 answers
22 views

Neural Network with numerical data and sentences as features

I'm beginning in the words A.I. features. My current problem is that I want to create Neural Network that takes as input numerical data and also words as data (by words, I mean multiple sentences) to ...
user avatar
1 vote
2 answers
38 views

Given embedding vector A and vector B, how to find top k embedding vectors such that they are similar to vector A and dissimilar to vector B

Which would be better approach for getting top k embedding vectors such that they are similar to embedding vector A and dissimilar to vector B. Approach 1: calculate ...
user avatar
  • 11
0 votes
0 answers
27 views

What are the types of inputs used for RNN in literature given sentences?

Suppose there are $m$ sentences in a text file and the number of distinct words is equal to $n$. The goal is to get word embeddings using RNN. We know that it is impossible to pass any word, which is ...
user avatar
  • 3,099
1 vote
1 answer
198 views

What exactly is embedding layer used in RNN encoders?

I am reading about RNN encoders. I came across the following line from this code. And I am facing difficulty in understanding the theoretical details regarding it. ...
user avatar
  • 3,099
1 vote
1 answer
46 views

General approaches in text encoding and labelling for NLP [closed]

What are the approaches of encoding text data? I would be glad to hear some summarization from experienced persons. And are there any solutions accepting words outside the vocabulary and including ...
user avatar
  • 111
0 votes
0 answers
17 views

How to improve the PMI (Pointwise Mutual Information) Quality for document based PMI

Generating word embeddings from the PMI is well understood and known to be equivalent to SGNS (skipgram negative-sampling) under certain conditions. I was able to get good quality word embedding using ...
user avatar
  • 101
3 votes
1 answer
97 views

Why do we multipy context_size with embedding_dim? (PyTorch)

I've been using Tensorflow and just started learning PyTorch. I was following the tutorial: https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html#sphx-glr-beginner-nlp-word-...
user avatar
  • 145
0 votes
0 answers
75 views

Bert vs Sentence-Bert

I read a paper about Rumor detection and they used BERT as an unsupervised language representation, fine-tuning it using a small dataset, and combining it with a supervised learning model to provide ...
user avatar
1 vote
2 answers
105 views

Can I always use "encoding" and "embedding" interchangeably?

This question is restricted to the text domain only. The meaning of the word "encode" is Convert (information or instruction) into a particular form. One which performs encoding is called an ...
user avatar
  • 3,099
0 votes
0 answers
3 views

Ensure trained word embeddings get high similarity with particular words

I am trying out my hand at training a Word2Vec model using gensim. I made a simple training file that basically had just one line repeated multiple times ...
user avatar
0 votes
0 answers
34 views

Keras MLP performing better than Transformers

I'm working on a comparative study using some models in a sentiment analysis task: MLPs and LSTMs with and without the use of word embeddings (GloVe and Word2Vec) and two Transformer-based models (...
user avatar
1 vote
0 answers
26 views

Why can't recurrent neural network handle large corpus for obtaining embeddings?

In order to learn the embeddings, we need to train a model based on some objective function. The model can be an RNN and the objective function can be the likelihood. We learn the embeddings by ...
user avatar
  • 3,099
1 vote
0 answers
41 views

How to generate text descriptions from keywords?

I wonder how can I build a neural network which will generate text description from given tag/tags. Let's assume I have created such data structure: ...
user avatar
0 votes
2 answers
155 views

What is the exact difference between distributional semantics and distributed semantics?

While studying word embeddings in natural language processing, I encountered the following statement on page 327 of the textbook Natural Language Processing by Jacob Eisenstein Distributional ...
user avatar
  • 3,099
1 vote
1 answer
69 views

Is categorical encoding a type of word embedding?

Word embedding refers to the techniques in which a word is represented by a vector. There are also integer encoding and one-hot encoding, which I will collectively call categorical encoding. I can see ...
user avatar
  • 3,099
1 vote
0 answers
28 views

Why are BERT embeddings interpreted as representations of the corresponding words?

It's often assumed in literature that BERT embeddings are contextual representations of the corresponding word. That is, if the 5th word is "cold", then the 5th BERT embedding is a ...
user avatar
1 vote
0 answers
41 views

How do sparse word embeddings fail to capture synonymy?

While reading some explanations of why dense word embeddings work better than sparse word embeddings, the following statement has been given in the chapter Vector Semantics and Embeddings, showing a ...
user avatar
  • 3,099
5 votes
2 answers
327 views

Is an embedding a representation of a word or its meaning?

What does the term "embedding" actually mean? An embedding is a vector, but is that vector a representation of a word or its meaning? Literature loosely uses the word for both purposes. ...
user avatar
  • 3,099
0 votes
2 answers
89 views

Book(s) for text embedding

Text here refers to either character or word or sentence. Is there any recent textbook that encompasses from classical methods to the modern techniques for embedding texts? If a single textbook is ...
user avatar
  • 3,099
1 vote
2 answers
347 views

Should I need to use BERT embeddings while tokenizing using BERT tokenizer?

I am new to BERT and NLP and I am a little confused with tokenization and word embedding. My doubt is if I use the BertTokenizer for tokenizing a sentence then do I have to compulsorily use ...
user avatar
4 votes
2 answers
2k views

What is the difference between a language model and a word embedding?

I am self-studying applications of deep learning on the NLP and machine translation. I am confused about the concepts of "Language Model", "Word Embedding", "BLEU Score". ...
user avatar
1 vote
0 answers
24 views

Given the word embeddings, how do I create the sentence composed of the corresponding words?

I have done some reading. I want to implement an LSTM with pre-trained word embeddings (I also have plans to create my word embeddings, but let's cross that bridge when we come to it). In any given ...
user avatar
  • 11
0 votes
1 answer
129 views

Are the Word2Vec encoded embeddings available online? [closed]

I am trying to do an NLP project and was wondering if there is anywhere online where the Word2Vec embeddings are stored (the actual n-dimmensional vectors). I want to search up a word and see what its ...
user avatar
0 votes
1 answer
51 views

NLP: Are hashtags tokenised?

I am exploring a potential NLP project. I was wondering what generally is done with the hashtags words (e.g. #hello). Are those words ignored? is the ...
user avatar
1 vote
0 answers
90 views

What does the outputlayer of BERT for masked language modelling look like?

In the tutorial BERT – State of the Art Language Model for NLP the masked language modeling pre-training steps are described as follows: In technical terms, the prediction of the output words ...
user avatar
9 votes
3 answers
2k views

What kind of word embedding is used in the original transformer?

I am currently trying to understand transformers. To start, I read Attention Is All You Need and also this tutorial. What makes me wonder is the word embedding used in the model. Is word2vec or GloVe ...
user avatar
0 votes
0 answers
37 views

Is there a reason why no one combines word embeddings with the median?

Could you combine word embeddings with the median per dimension to get a document embedding? In my case I have a huge amount of words to build one document, which in turn should describe a topic. I ...
user avatar
2 votes
0 answers
40 views

Can One-Hot Vectors be used as Inputs for Recurrent Neural Networks?

When using an RNN to encode a sentence, one normally takes each word, passes it through an embedding layer, and then uses the dense embedding as the input into the RNN. Lets say instead of using dense ...
user avatar
1 vote
2 answers
177 views

Why is an embedding of dimension 400 enough to represent 70000 words?

I am learning PyTorch on Udacity. In lesson 8, section 11: Training the Model, the instructor writes: Then I have my embedding and hidden dimension. The embedding dimension is just a smaller ...
user avatar
  • 181
2 votes
1 answer
165 views

Is there a pretrained (NLP) transformer that uses subword n-gram embeddings for tokenization like fasttext?

I know that several tokenization methods that are used for tranformer models like WordPiece for Bert and BPE for Roberta and others. What I was wondering if there is also a transformer which uses a ...
user avatar
  • 21
1 vote
0 answers
17 views

Bechmark models for Text Classification / Sentiment Classification

I am currently working on a novel application in NLP where I try to classify empathic and non-empathic texts. I would like to compare the performance of my model to some benchmark models. As I am ...
user avatar
  • 11
1 vote
0 answers
141 views

How homographs is an NLP task can be treated?

A homograph - is a word that shares the same written form as another word but has a different meaning. They can be even different parts of speech. For example: close(verb) - close(adverb) lead(verb)...
user avatar
3 votes
3 answers
95 views

When to convert data to word embeddings in NLP

When training a network using word embeddings, it is standard to add an embedding layer to first convert the input vector to the embeddings. However, assuming the embeddings are pre-trained and frozen,...
user avatar
1 vote
1 answer
2k views

How is dropout applied to the embedding layer's output?

...
user avatar
  • 197
1 vote
0 answers
80 views

Is it good practice to save NLP Transformer based pre-trained models into file system in production environment

I have developed a multi label classifier using BERT. I'm leveraging Hugging Face Pytorch implementation for transformers. I have saved the pretrained model into the file directory in dev environment. ...
user avatar
  • 141
3 votes
1 answer
65 views

How can I create an embedding layer to convert words to a vector space from scratch?

For an upcoming project, I am trying to build a neural network for classifying text from scratch, without the use of libraries. This requires an embedding layer, or a way to convert words to some ...
user avatar
0 votes
1 answer
116 views

How to add a pretrained model to my layers to get embeddings?

I want to use a pretrained model found in [BERT Embeddings] https://github.com/UKPLab/sentence-transformers and I want to add a layer to get the sentence embeddings from the model and pass on to the ...
user avatar
1 vote
0 answers
35 views

How many spectrogram frames per input character does text-to-speech (TTS) system Tacotron-2 generate?

I've been reading on Tacotron-2, a text-to-speech system, that generates speech just-like humans (indistinguishable from humans) using the GitHub https://github.com/Rayhane-mamah/Tacotron-2. I'm very ...
user avatar
  • 151
3 votes
1 answer
44 views

What should the dimension of the input be for text summarization?

I am trying to build a model for extractive text summarization using keras sequential layers. I am having a hard time trying to understand how to input my x data. Should it be an array of documents ...
user avatar
1 vote
0 answers
52 views

Creating Text Features using word2vec

My task is to classify some texts. I have used word2vec to represent text words and I pass them to an LSTM as input. Taking into account that texts do not contain the same number of words, is it a ...
user avatar
  • 889
3 votes
0 answers
86 views

Is there a good book or paper on word embeddings?

Is there a good and modern book that focuses on word embeddings and their applications? It would also be ok to provide the name of a paper that provides a good overview of word embeddings.
user avatar
  • 889
1 vote
1 answer
39 views

Why I have a different number of terms in word2vec and TFIDF? How I can fix it?

I need multiply the weigths of terms in TFIDF matrix by the word-embeddings of word2vec matrix but I can't do it because each matrix have a different number of terms. I am using the same corpus for ...
user avatar
  • 13
1 vote
1 answer
53 views

Using word embedding to extend words for searching POI names

I am developing my own mobile app related to digital map. One of the functions is searching POIs (points of interest) in the map according to relevance between user query and POI name. Besides the ...
user avatar
  • 111
1 vote
0 answers
25 views

Is there a way to parallelize GloVe cooccur function?

I would like to create a GloVe word embedding on a very large corpus (trillions of words). However, creating the co-occurence matrix with the GloVe cooccur script is projected to take weeks. Is there ...
user avatar
4 votes
2 answers
106 views

Does summing up word vectors destroy their meaning?

For example, I have a paragraph that I want to classify in a binary manner. But because the inputs have to have a fixed length, I need to ensure that every paragraph is represented by a uniform ...
user avatar
8 votes
2 answers
3k views

Is word embedding a form of feature extraction?

Feature extraction is a concept concerning the translation of raw data into the inputs that a particular machine learning algorithm requires. These derived features from the raw data that are actually ...
user avatar
  • 83
2 votes
0 answers
61 views

How can I feed any word into a neural network?

I am working on an Intent detection problem for a chatbot in Java. So I need to convert words from String to a double[] format. I tried using wordToVec(deeplearning4j), but it does not return a vector ...
user avatar
3 votes
0 answers
36 views

Why embedding layer is used in the character-level Natural Language Processing models

Problem Background I am working with a problem, which requires a character-level, deep learning model. Previously I was working with word-level deep NLP (Natural Language Processing) models, and in ...
user avatar