Questions tagged [word-embedding]

For questions related to word embeddings, which are vector representations of words.

Filter by
Sorted by
Tagged with
1
vote
0answers
25 views

How homographs is an NLP task can be treated?

A homograph - is a word that shares the same written form as another word but has a different meaning. They can be even different parts of speech. For example: close(verb) - close(adverb) lead(verb)...
2
votes
3answers
57 views

When to convert data to word embeddings in NLP

When training a network using word embeddings, it is standard to add an embedding layer to first convert the input vector to the embeddings. However, assuming the embeddings are pre-trained and frozen,...
0
votes
0answers
12 views

Automated Scoring (non-english language) Using BERT

i'm a student and i'm new to NLP. I want to build an Automated Scoring system which is in Indonesian Language using BERT. The system is expected to be able to measure the similarity of an answer(e.g: ...
1
vote
0answers
43 views

Is it good practice to save NLP Transformer based pre-trained models into file system in production environment

I have developed a multi label classifier using BERT. I'm leveraging Hugging Face Pytorch implementation for transformers. I have saved the pretrained model into the file directory in dev environment. ...
1
vote
1answer
33 views

How can I create an embedding layer to convert words to a vector space from scratch?

For an upcoming project, I am trying to build a neural network for classifying text from scratch, without the use of libraries. This requires an embedding layer, or a way to convert words to some ...
0
votes
1answer
39 views

How to add a pretrained model to my layers to get embeddings?

I want to use a pretrained model found in [BERT Embeddings] https://github.com/UKPLab/sentence-transformers and I want to add a layer to get the sentence embeddings from the model and pass on to the ...
1
vote
0answers
16 views

How many spectrogram frames per input character does text-to-speech (TTS) system Tacotron-2 generate?

I've been reading on Tacotron-2, a text-to-speech system, that generates speech just-like humans (indistinguishable from humans) using the GitHub https://github.com/Rayhane-mamah/Tacotron-2. I'm very ...
3
votes
1answer
35 views

What should the dimension of the input be for text summarization?

I am trying to build a model for extractive text summarization using keras sequential layers. I am having a hard time trying to understand how to input my x data. Should it be an array of documents ...
0
votes
0answers
22 views

What is the right way to set the dimension of the word representation in SkipGram word2vec?

I know that in word2vec each word has two word representations; one for the center word and one for the context word. What is the right way to set the dimension of the center word using SkipGram, ...
1
vote
0answers
30 views

Creating Text Features using word2vec

My task is to classify some texts. I have used word2vec to represent text words and I pass them to an LSTM as input. Taking into account that texts do not contain the same number of words, is it a ...
0
votes
0answers
30 views

Which word representation should I use in word2vec?

I know that in word2vec each word has two word representations; one for center word and one for context word. In NLP, when I want to set the words of a text as an input of an LSTM/RNN using word2vec, ...
2
votes
0answers
56 views

Is there a good book or paper on word embeddings?

Is there a good and modern book that focuses on word embeddings and their applications? It would also be ok to provide the name of a paper that provides a good overview of word embeddings.
1
vote
0answers
51 views

How get matrix of word embeddings in FastText of gensim?

I try get the matrix embedding of my model but I can't because although this code gives no error it never ends running. The code is: ...
1
vote
1answer
29 views

Why I have a different number of terms in word2vec and TFIDF? How I can fix it?

I need multiply the weigths of terms in TFIDF matrix by the word-embeddings of word2vec matrix but I can't do it because each matrix have a different number of terms. I am using the same corpus for ...
1
vote
1answer
28 views

Using word embedding to extend words for searching POI names

I am developing my own mobile app related to digital map. One of the functions is searching POIs (points of interest) in the map according to relevance between user query and POI name. Besides the ...
1
vote
0answers
22 views

Is there a way to parallelize GloVe cooccur function?

I would like to create a GloVe word embedding on a very large corpus (trillions of words). However, creating the co-occurence matrix with the GloVe cooccur script is projected to take weeks. Is there ...
6
votes
2answers
663 views

Is word embedding a form of feature extraction?

Feature extraction is a concept concerning the translation of raw data into the inputs that a particular machine learning algorithm requires. These derived features from the raw data that are actually ...
2
votes
0answers
38 views

How can I feed any word into a neural network?

I am working on an Intent detection problem for a chatbot in Java. So I need to convert words from String to a double[] format. I tried using wordToVec(deeplearning4j), but it does not return a vector ...
3
votes
0answers
28 views

Why embedding layer is used in the character-level Natural Language Processing models

Problem Background I am working with a problem, which requires a character-level, deep learning model. Previously I was working with word-level deep NLP (Natural Language Processing) models, and in ...
0
votes
0answers
16 views

Why word embedding such as word2vec is not used as the output layer of a seq2seq decoder?

It would make sense to make the decoder predict a smaller embedding vector instead of softmax over a large dictionary. The word having the most cosine similarity with the output embedding could be ...
1
vote
2answers
77 views

Why is embedding important in NLP, and how does autoencoder work?

People say embedding is necessary in NLP because if using just the word indices, the efficiency is not high as similar words are supposed to be related to each other. However, I still don't truly get ...
2
votes
0answers
17 views

Doubt on formulating cost function for GloVe

I'm reading the notes here and have a doubt on page 2 ("Least squares objective" section). The probability of a word $j$ occurring in the context of word $i$ is $$Q_{ij}=\frac{\exp(u_j^Tv_i)}{\sum_{w=...
0
votes
1answer
67 views

Real time ticket similarity

I'm dealing with a "ticket similarity task". Every time new tickets arrive at the help desk (customer service), I need to compare them and find out about similar ones. In this way, once the operator ...
1
vote
0answers
33 views

Reference request: one-hot encoding outperforming random orthogonal encoding

I experimented with a CNN operating on texts encoded as sequences of character vectors, where characters are encoded as one-hot vectors in one embedding and as random unit length pairwise orthogonal ...
3
votes
1answer
63 views

How does Continuous Bag of Words ensure that similar words are encoded as similar embeddings?

This is related to my earlier question, which I'm trying to break down into parts (this being the first). I'm reading notes on word vectors here. Specifically, I'm referring to section 4.2 on page 7. ...
0
votes
0answers
21 views

Understanding how continuous bag of words method learns embedded representations

I'm reading notes on word vectors here. Specifically, I'm referring to section 4.2 on page 7. First, regarding points 1 to 6 - here's my understanding: If we have a vocabulary $V$, the naive way to ...
2
votes
2answers
283 views

Can ELMO embeddings be used to find the n most similar sentences?

Assume I have a list of sentences, which is just a list of strings. I need a way of comparing some input string against those sentences to find the most similar. Can ELMO embeddings be used to train a ...
2
votes
0answers
124 views

How does FastText support online learning?

I'm using FastText pre-trained-embedding for tackling a classification task, but I saw it supports also online training (incremental training) for adding domain-specific corpus. How does it work? ...
2
votes
1answer
4k views

Adding BERT embeddings in LSTM embedding layer

I am planning to use BERT embeddings in the LSTM embedding layer instead of the usual Word2vec/Glove Embeddings. What are the possible ways to do that?
4
votes
1answer
97 views

Will BERT embedding be always same for a given document when used as a feature extractor

When we use BERT embeddings for a classification task, would we get different embeddings every time we pass the same text through the BERT architecture? If yes, is it the right way to use the ...
1
vote
0answers
79 views

Multiple embedding layers?

How would one go about inputting multiple high dimensionality categorical columns using TensorFlow's Embedding Feature Columns? Does that even make sense to do? For example: for a car price predictor,...
1
vote
0answers
19 views

Creating a zero element in embedding space

I have some variable length input vectors for my own use case of a 'stylistic transfer'-esque process, and I am wondering if anyone knows of a way to engineer an input that maps to a 0 element in ...
2
votes
2answers
62 views

Do we have cross-language vector space for word embedding?

Do we have cross-language vector space for word embedding? When measure similarity for apple/Pomme/mela/Lacus/苹果/りんご, they should be the same If would be great if there's available internet service ...
1
vote
1answer
52 views

How can we create a vector space where word spelling and pronunciation can be easily compared?

In natural language processing, we can convert words to vectors (or word embeddings). In this vector space, we can measure the similarity between these word embeddings. How can we create a vector ...
3
votes
0answers
24 views

Why does all of NLP literature use noise contrastive estimation loss for negative sampling instead of sampled softmax loss?

A sampled softmax function is like a regular softmax but randomly selects a given number of 'negative' samples. This is difference than NCE Loss, which doesn't use a softmax at all, it uses a ...
1
vote
1answer
67 views

Skip-Gram Model Training

Suppose we want to predict context words $w_{i-h}, \dots, w_{i+h}$ given a target word $w_i$ for a window size $h$ around the target word $w_i$. We can represent this as: $$p(w_{i-h}, \dots, w_{i+h}|...
1
vote
1answer
118 views

What do the vectors of the center and outside word look like in word2vec?

In word2vec, the task is to learn to predict which words are most likely to be near each other in some long corpus of text. For each word $c$ in the corpus, the model outputs the probability ...
1
vote
0answers
46 views

How could I learn tree paths given word embeddings?

I need to map from a vector space representation onto a tree structure. A possible solution: given a word vector as input, produce a path in the tree from the root down to the node that most closely ...
3
votes
2answers
119 views

How should the output layer of an LSTM be when the output are word embeddings?

I'm having trouble grasping how to output word embeddings from an LSTM model. I'm seeing many examples using a softmax activation function on the output, but for that I would need to output one hot ...
4
votes
1answer
48 views

Do individual dimensions in vector space have meaning?

Word2vec assigns an N-dimensional vector to given words (which can be considered a form of dimensionality reduction). It turns out that, at least with a number of canonical examples, vector ...
2
votes
2answers
67 views

Intuition on how word embeddings bring information to a network

How is it that word embedding layer (say word2vec) brings more insights to the network compared to a simple one hot encoded layer? I understand how word embedding carry some semantic meaning, but it ...
1
vote
1answer
92 views

How is the word embedding represented in the paper “Recurrent neural network based language model”?

I'm reading "Recurrent neural network based language model" of Mikolov et al. (2010). Although the article is straight forward, I'm not sure how word embedding $w(t)$ is obtained: The reason I wonder ...
-1
votes
1answer
50 views

Over-exposure of certain items in content based recommendation engine

I'm working on a content based recommendation engine for ebooks. I create document vectors with 300 features for every ebook using a word2vec model trained on google news and determine recommendations ...
2
votes
0answers
69 views

How can we build word embeddings for a language? [closed]

Which algorithms are there to create word embeddings for a given language?