Questions tagged [word-embedding]

For questions related to word embeddings, which are vector representations of words.

Filter by
Sorted by
Tagged with
0
votes
0answers
3 views

Ensure trained word embeddings get high similarity with particular words

I am trying out my hand at training a Word2Vec model using gensim. I made a simple training file that basically had just one line repeated multiple times ...
0
votes
0answers
20 views

Keras MLP performing better than Transformers

I'm working on a comparative study using some models in a sentiment analysis task: MLPs and LSTMs with and without the use of word embeddings (GloVe and Word2Vec) and two Transformer-based models (...
1
vote
0answers
22 views

Why can't recurrent neural network handle large corpus for obtaining embeddings?

In order to learn the embeddings, we need to train a model based on some objective function. The model can be an RNN and the objective function can be the likelihood. We learn the embeddings by ...
1
vote
0answers
23 views

How to generate text descriptions from keywords?

I wonder how can I build a neural network which will generate text description from given tag/tags. Let's assume I have created such data structure: ...
0
votes
0answers
24 views

How to perform negative sampling in Word2Vec?

Word2Vec is an algorithm that generates word embeddings using logistic regression. Positive and negative examples are needed for logistic regression. Word2Vec uses the neighboring words that are ...
0
votes
2answers
40 views

What is the exact difference between distributional semantics and distributed semantics?

While studying word embeddings in natural language processing, I encountered the following statement on page 327 of the textbook Natural Language Processing by Jacob Eisenstein Distributional ...
1
vote
1answer
43 views

Is categorical encoding a type of word embedding?

Word embedding refers to the techniques in which a word is represented by a vector. There are also integer encoding and one-hot encoding, which I will collectively call categorical encoding. I can see ...
1
vote
0answers
20 views

Why are BERT embeddings interpreted as representations of the corresponding words?

It's often assumed in literature that BERT embeddings are contextual representations of the corresponding word. That is, if the 5th word is "cold", then the 5th BERT embedding is a ...
1
vote
0answers
32 views

Do literature uses (text) encoding and (word) embedding in a loose sense?

Many research papers use the word "encoding" in many ways. One way is that they use the word "encoding" for models that do convert the text into a vector (say text encoding or ...
1
vote
0answers
32 views

How do sparse word embeddings fail to capture synonymy?

While reading some explanations of why dense word embeddings work better than sparse word embeddings, the following statement has been given in the chapter Vector Semantics and Embeddings, showing a ...
5
votes
2answers
294 views

Is an embedding a representation of a word or its meaning?

What does the term "embedding" actually mean? An embedding is a vector, but is that vector a representation of a word or its meaning? Literature loosely uses the word for both purposes. ...
0
votes
2answers
78 views

Book(s) for text embedding

Text here refers to either character or word or sentence. Is there any recent textbook that encompasses from classical methods to the modern techniques for embedding texts? If a single textbook is ...
0
votes
0answers
19 views

Word2vec - CBOW and Skipgram comparative study for big data

Word2vec - CBOW and Skipgram comparative study As a part of my thesis I am creating a comparative study of CBOW and Skipgram for big data. My input is a Wikipedia dump and I have created a single .txt ...
0
votes
0answers
16 views

Adding BERT embeddings in BiLSTM embedding layer

I am want to use BERT embeddings in the BiLSTM embedding layer instead of Word2Vec or FastText Embeddings. There is any code to do that?
0
votes
1answer
50 views

Should I need to use BERT embeddings while tokenizing using BERT tokenizer?

I am new to BERT and NLP and I am a little confused with tokenization and word embedding. My doubt is if I use the BertTokenizer for tokenizing a sentence then do I have to compulsorily use ...
0
votes
0answers
21 views

How does Byte Pair Embedding affect the effectiveness of embedding techniques like Word2Vec?

I see Byte Pair Embedding (BPE) is frequently adopted for NLP to reduce the vocabulary size. How does it affect the effectiveness of embedding techniques like Word2Vec?
3
votes
2answers
331 views

What is the difference between a language model and a word embedding?

I am self-studying applications of deep learning on the NLP and machine translation. I am confused about the concepts of "Language Model", "Word Embedding", "BLEU Score". ...
1
vote
0answers
21 views

Given the word embeddings, how do I create the sentence composed of the corresponding words?

I have done some reading. I want to implement an LSTM with pre-trained word embeddings (I also have plans to create my word embeddings, but let's cross that bridge when we come to it). In any given ...
0
votes
0answers
16 views

Can an Autoencoder be used to create a simple text-completion plugin (code-completion for programming)?

Can an Autoencoder neural network be used to create a simple code completion plugin for a developer using a certain programming language ? The idea is that the training data will be generated from ...
0
votes
1answer
32 views

Are the Word2Vec encoded embeddings available online? [closed]

I am trying to do an NLP project and was wondering if there is anywhere online where the Word2Vec embeddings are stored (the actual n-dimmensional vectors). I want to search up a word and see what its ...
0
votes
1answer
25 views

NLP: Are hashtags tokenised?

I am exploring a potential NLP project and was wondering what generally is done with the hashtags words (e.g. #hello). Are those words ignored? is the # removed and the word tokenised? is it tokenised ...
1
vote
0answers
28 views

What does the outputlayer of BERT for masked language modelling look like?

In the tutorial BERT – State of the Art Language Model for NLP the masked language modeling pre-training steps are described as follows: In technical terms, the prediction of the output words ...
0
votes
0answers
37 views

Seq2Seq Modelling: when implementing some machine translation net, how are special tokens embedded?

When implementing any encoder-decoder network for machine translation, during training we provide the true output sentence to the decoder so that the context vector (from source language) may be ...
2
votes
3answers
118 views

What kind of word embedding is used in the original transformer?

I am currently trying to understand transformers. To start, I read Attention Is All You Need and also this tutorial. What makes me wonder is the word embedding used in the model. Is word2vec or GloVe ...
0
votes
0answers
34 views

Is there a reason why no one combines word embeddings with the median?

Could you combine word embeddings with the median per dimension to get a document embedding? In my case I have a huge amount of words to build one document, which in turn should describe a topic. I ...
0
votes
0answers
18 views

How to chose a specific contextual embedding?

I am learning deep learning using a plethora of online resources. I have been using mostly word2vec and gloVe for my NMT model. I recently came across the concept of contextual-embedding, and I see ...
2
votes
0answers
38 views

Can One-Hot Vectors be used as Inputs for Recurrent Neural Networks?

When using an RNN to encode a sentence, one normally takes each word, passes it through an embedding layer, and then uses the dense embedding as the input into the RNN. Lets say instead of using dense ...
1
vote
2answers
57 views

Why is an embedding of dimension 400 enough to represent 70000 words?

I am learning PyTorch on Udacity. In lesson 8, section 11: Training the Model, the instructor writes: Then I have my embedding and hidden dimension. The embedding dimension is just a smaller ...
0
votes
0answers
27 views

Is there a pretrained (NLP) transformer that uses subword n-gram embeddings for tokenization like fasttext?

I know that several tokenization methods that are used for tranformer models like WordPiece for Bert and BPE for Roberta and others. What I was wondering if there is also a transformer which uses a ...
1
vote
0answers
17 views

Bechmark models for Text Classification / Sentiment Classification

I am currently working on a novel application in NLP where I try to classify empathic and non-empathic texts. I would like to compare the performance of my model to some benchmark models. As I am ...
1
vote
0answers
62 views

How homographs is an NLP task can be treated?

A homograph - is a word that shares the same written form as another word but has a different meaning. They can be even different parts of speech. For example: close(verb) - close(adverb) lead(verb)...
3
votes
3answers
79 views

When to convert data to word embeddings in NLP

When training a network using word embeddings, it is standard to add an embedding layer to first convert the input vector to the embeddings. However, assuming the embeddings are pre-trained and frozen,...
0
votes
0answers
24 views

Automated Scoring (non-english language) Using BERT

i'm a student and i'm new to NLP. I want to build an Automated Scoring system which is in Indonesian Language using BERT. The system is expected to be able to measure the similarity of an answer(e.g: ...
1
vote
0answers
75 views

Is it good practice to save NLP Transformer based pre-trained models into file system in production environment

I have developed a multi label classifier using BERT. I'm leveraging Hugging Face Pytorch implementation for transformers. I have saved the pretrained model into the file directory in dev environment. ...
3
votes
1answer
49 views

How can I create an embedding layer to convert words to a vector space from scratch?

For an upcoming project, I am trying to build a neural network for classifying text from scratch, without the use of libraries. This requires an embedding layer, or a way to convert words to some ...
0
votes
1answer
58 views

How to add a pretrained model to my layers to get embeddings?

I want to use a pretrained model found in [BERT Embeddings] https://github.com/UKPLab/sentence-transformers and I want to add a layer to get the sentence embeddings from the model and pass on to the ...
1
vote
0answers
23 views

How many spectrogram frames per input character does text-to-speech (TTS) system Tacotron-2 generate?

I've been reading on Tacotron-2, a text-to-speech system, that generates speech just-like humans (indistinguishable from humans) using the GitHub https://github.com/Rayhane-mamah/Tacotron-2. I'm very ...
3
votes
1answer
41 views

What should the dimension of the input be for text summarization?

I am trying to build a model for extractive text summarization using keras sequential layers. I am having a hard time trying to understand how to input my x data. Should it be an array of documents ...
1
vote
0answers
42 views

Creating Text Features using word2vec

My task is to classify some texts. I have used word2vec to represent text words and I pass them to an LSTM as input. Taking into account that texts do not contain the same number of words, is it a ...
3
votes
0answers
73 views

Is there a good book or paper on word embeddings?

Is there a good and modern book that focuses on word embeddings and their applications? It would also be ok to provide the name of a paper that provides a good overview of word embeddings.
1
vote
0answers
184 views

How get matrix of word embeddings in FastText of gensim?

I try get the matrix embedding of my model but I can't because although this code gives no error it never ends running. The code is: ...
1
vote
1answer
33 views

Why I have a different number of terms in word2vec and TFIDF? How I can fix it?

I need multiply the weigths of terms in TFIDF matrix by the word-embeddings of word2vec matrix but I can't do it because each matrix have a different number of terms. I am using the same corpus for ...
1
vote
1answer
34 views

Using word embedding to extend words for searching POI names

I am developing my own mobile app related to digital map. One of the functions is searching POIs (points of interest) in the map according to relevance between user query and POI name. Besides the ...
1
vote
0answers
23 views

Is there a way to parallelize GloVe cooccur function?

I would like to create a GloVe word embedding on a very large corpus (trillions of words). However, creating the co-occurence matrix with the GloVe cooccur script is projected to take weeks. Is there ...
8
votes
2answers
2k views

Is word embedding a form of feature extraction?

Feature extraction is a concept concerning the translation of raw data into the inputs that a particular machine learning algorithm requires. These derived features from the raw data that are actually ...
2
votes
0answers
47 views

How can I feed any word into a neural network?

I am working on an Intent detection problem for a chatbot in Java. So I need to convert words from String to a double[] format. I tried using wordToVec(deeplearning4j), but it does not return a vector ...
3
votes
0answers
30 views

Why embedding layer is used in the character-level Natural Language Processing models

Problem Background I am working with a problem, which requires a character-level, deep learning model. Previously I was working with word-level deep NLP (Natural Language Processing) models, and in ...
1
vote
2answers
146 views

Why is embedding important in NLP, and how does autoencoder work?

People say embedding is necessary in NLP because if using just the word indices, the efficiency is not high as similar words are supposed to be related to each other. However, I still don't truly get ...
2
votes
0answers
20 views

Doubt on formulating cost function for GloVe

I'm reading the notes here and have a doubt on page 2 ("Least squares objective" section). The probability of a word $j$ occurring in the context of word $i$ is $$Q_{ij}=\frac{\exp(u_j^Tv_i)}{\sum_{w=...