Questions tagged [word-embedding]

For questions related to word embeddings, which are vector representations of words.

Filter by
Sorted by
Tagged with
-1
votes
0answers
9 views

Word2Vec: Identifying many-to-one relationships between words

Standard introductory examples in Word2Vec, like king - queen = man - woman and tokyo - japan = london - uk, involve one-to-one ...
0
votes
0answers
12 views

Adding BERT embeddings in BiLSTM embedding layer

I am want to use BERT embeddings in the BiLSTM embedding layer instead of Word2Vec or FastText Embeddings. There is any code to do that?
0
votes
1answer
26 views

Should I need to use BERT embeddings while tokenizing using BERT tokenizer?

I am new to BERT and NLP and I am a little confused with tokenization and word embedding. My doubt is if I use the BertTokenizer for tokenizing a sentence then do I have to compulsorily use ...
0
votes
0answers
17 views

How does Byte Pair Embedding affect the effectiveness of embedding techniques like Word2Vec?

I see Byte Pair Embedding (BPE) is frequently adopted for NLP to reduce the vocabulary size. How does it affect the effectiveness of embedding techniques like Word2Vec?
1
vote
2answers
45 views

What is the difference between a language model and a word embedding?

I am self-studying applications of deep learning on the NLP and machine translation. I am confused about the concepts of "Language Model", "Word Embedding", "BLEU Score". ...
1
vote
0answers
20 views

Given the word embeddings, how do I create the sentence composed of the corresponding words?

I have done some reading. I want to implement an LSTM with pre-trained word embeddings (I also have plans to create my word embeddings, but let's cross that bridge when we come to it). In any given ...
0
votes
0answers
15 views

Can an Autoencoder be used to create a simple text-completion plugin (code-completion for programming)?

Can an Autoencoder neural network be used to create a simple code completion plugin for a developer using a certain programming language ? The idea is that the training data will be generated from ...
0
votes
1answer
25 views

Are the Word2Vec encoding available online [closed]

I am trying to do an NLP project and was wondering if there is anywhere online where the Word2Vec encoding are stored. I want to search up a word and see what its encoding is. I have tried looking but ...
0
votes
1answer
24 views

NLP: Are hashtags tokenised?

I am exploring a potential NLP project and was wondering what generally is done with the hashtags words (e.g. #hello). Are those words ignored? is the # removed and the word tokenised? is it tokenised ...
1
vote
0answers
26 views

What does the outputlayer of BERT for masked language modelling look like?

In the tutorial BERT – State of the Art Language Model for NLP the masked language modeling pre-training steps are described as follows: In technical terms, the prediction of the output words ...
0
votes
0answers
34 views

Seq2Seq Modelling: when implementing some machine translation net, how are special tokens embedded?

When implementing any encoder-decoder network for machine translation, during training we provide the true output sentence to the decoder so that the context vector (from source language) may be ...
1
vote
2answers
54 views

What kind of word embedding is used in the original transformer?

I am currently trying to understand transformers. To start, I read Attention Is All You Need and also this tutorial. What makes me wonder is the word embedding used in the model. Is word2vec or GloVe ...
0
votes
0answers
34 views

Is there a reason why no one combines word embeddings with the median?

Could you combine word embeddings with the median per dimension to get a document embedding? In my case I have a huge amount of words to build one document, which in turn should describe a topic. I ...
0
votes
0answers
16 views

How to chose a specific contextual embedding?

I am learning deep learning using a plethora of online resources. I have been using mostly word2vec and gloVe for my NMT model. I recently came across the concept of contextual-embedding, and I see ...
2
votes
0answers
37 views

Can One-Hot Vectors be used as Inputs for Recurrent Neural Networks?

When using an RNN to encode a sentence, one normally takes each word, passes it through an embedding layer, and then uses the dense embedding as the input into the RNN. Lets say instead of using dense ...
1
vote
2answers
48 views

Why is an embedding of dimension 400 enough to represent 70000 words?

I am learning PyTorch on Udacity. In lesson 8, section 11: Training the Model, the instructor writes: Then I have my embedding and hidden dimension. The embedding dimension is just a smaller ...
0
votes
0answers
20 views

Is there a pretrained (NLP) transformer that uses subword n-gram embeddings for tokenization like fasttext?

I know that several tokenization methods that are used for tranformer models like WordPiece for Bert and BPE for Roberta and others. What I was wondering if there is also a transformer which uses a ...
1
vote
0answers
17 views

Bechmark models for Text Classification / Sentiment Classification

I am currently working on a novel application in NLP where I try to classify empathic and non-empathic texts. I would like to compare the performance of my model to some benchmark models. As I am ...
1
vote
0answers
46 views

How homographs is an NLP task can be treated?

A homograph - is a word that shares the same written form as another word but has a different meaning. They can be even different parts of speech. For example: close(verb) - close(adverb) lead(verb)...
2
votes
3answers
73 views

When to convert data to word embeddings in NLP

When training a network using word embeddings, it is standard to add an embedding layer to first convert the input vector to the embeddings. However, assuming the embeddings are pre-trained and frozen,...
0
votes
0answers
23 views

Automated Scoring (non-english language) Using BERT

i'm a student and i'm new to NLP. I want to build an Automated Scoring system which is in Indonesian Language using BERT. The system is expected to be able to measure the similarity of an answer(e.g: ...
1
vote
0answers
62 views

Is it good practice to save NLP Transformer based pre-trained models into file system in production environment

I have developed a multi label classifier using BERT. I'm leveraging Hugging Face Pytorch implementation for transformers. I have saved the pretrained model into the file directory in dev environment. ...
3
votes
1answer
47 views

How can I create an embedding layer to convert words to a vector space from scratch?

For an upcoming project, I am trying to build a neural network for classifying text from scratch, without the use of libraries. This requires an embedding layer, or a way to convert words to some ...
0
votes
1answer
47 views

How to add a pretrained model to my layers to get embeddings?

I want to use a pretrained model found in [BERT Embeddings] https://github.com/UKPLab/sentence-transformers and I want to add a layer to get the sentence embeddings from the model and pass on to the ...
1
vote
0answers
22 views

How many spectrogram frames per input character does text-to-speech (TTS) system Tacotron-2 generate?

I've been reading on Tacotron-2, a text-to-speech system, that generates speech just-like humans (indistinguishable from humans) using the GitHub https://github.com/Rayhane-mamah/Tacotron-2. I'm very ...
3
votes
1answer
41 views

What should the dimension of the input be for text summarization?

I am trying to build a model for extractive text summarization using keras sequential layers. I am having a hard time trying to understand how to input my x data. Should it be an array of documents ...
0
votes
0answers
23 views

What is the right way to set the dimension of the word representation in SkipGram word2vec?

I know that in word2vec each word has two word representations; one for the center word and one for the context word. What is the right way to set the dimension of the center word using SkipGram, ...
1
vote
0answers
38 views

Creating Text Features using word2vec

My task is to classify some texts. I have used word2vec to represent text words and I pass them to an LSTM as input. Taking into account that texts do not contain the same number of words, is it a ...
2
votes
0answers
63 views

Is there a good book or paper on word embeddings?

Is there a good and modern book that focuses on word embeddings and their applications? It would also be ok to provide the name of a paper that provides a good overview of word embeddings.
1
vote
0answers
127 views

How get matrix of word embeddings in FastText of gensim?

I try get the matrix embedding of my model but I can't because although this code gives no error it never ends running. The code is: ...
1
vote
1answer
32 views

Why I have a different number of terms in word2vec and TFIDF? How I can fix it?

I need multiply the weigths of terms in TFIDF matrix by the word-embeddings of word2vec matrix but I can't do it because each matrix have a different number of terms. I am using the same corpus for ...
1
vote
1answer
30 views

Using word embedding to extend words for searching POI names

I am developing my own mobile app related to digital map. One of the functions is searching POIs (points of interest) in the map according to relevance between user query and POI name. Besides the ...
1
vote
0answers
23 views

Is there a way to parallelize GloVe cooccur function?

I would like to create a GloVe word embedding on a very large corpus (trillions of words). However, creating the co-occurence matrix with the GloVe cooccur script is projected to take weeks. Is there ...
7
votes
2answers
1k views

Is word embedding a form of feature extraction?

Feature extraction is a concept concerning the translation of raw data into the inputs that a particular machine learning algorithm requires. These derived features from the raw data that are actually ...
2
votes
0answers
43 views

How can I feed any word into a neural network?

I am working on an Intent detection problem for a chatbot in Java. So I need to convert words from String to a double[] format. I tried using wordToVec(deeplearning4j), but it does not return a vector ...
3
votes
0answers
30 views

Why embedding layer is used in the character-level Natural Language Processing models

Problem Background I am working with a problem, which requires a character-level, deep learning model. Previously I was working with word-level deep NLP (Natural Language Processing) models, and in ...
1
vote
2answers
120 views

Why is embedding important in NLP, and how does autoencoder work?

People say embedding is necessary in NLP because if using just the word indices, the efficiency is not high as similar words are supposed to be related to each other. However, I still don't truly get ...
2
votes
0answers
17 views

Doubt on formulating cost function for GloVe

I'm reading the notes here and have a doubt on page 2 ("Least squares objective" section). The probability of a word $j$ occurring in the context of word $i$ is $$Q_{ij}=\frac{\exp(u_j^Tv_i)}{\sum_{w=...
0
votes
1answer
96 views

Real time ticket similarity

I'm dealing with a "ticket similarity task". Every time new tickets arrive at the help desk (customer service), I need to compare them and find out about similar ones. In this way, once the operator ...
1
vote
0answers
51 views

Reference request: one-hot encoding outperforming random orthogonal encoding

I experimented with a CNN operating on texts encoded as sequences of character vectors, where characters are encoded as one-hot vectors in one embedding and as random unit length pairwise orthogonal ...
3
votes
1answer
69 views

How does Continuous Bag of Words ensure that similar words are encoded as similar embeddings?

This is related to my earlier question, which I'm trying to break down into parts (this being the first). I'm reading notes on word vectors here. Specifically, I'm referring to section 4.2 on page 7. ...
0
votes
0answers
21 views

Understanding how continuous bag of words method learns embedded representations

I'm reading notes on word vectors here. Specifically, I'm referring to section 4.2 on page 7. First, regarding points 1 to 6 - here's my understanding: If we have a vocabulary $V$, the naive way to ...
2
votes
2answers
543 views

Can ELMO embeddings be used to find the n most similar sentences?

Assume I have a list of sentences, which is just a list of strings. I need a way of comparing some input string against those sentences to find the most similar. Can ELMO embeddings be used to train a ...
2
votes
0answers
181 views

How does FastText support online learning?

I'm using FastText pre-trained-embedding for tackling a classification task, but I saw it supports also online training (incremental training) for adding domain-specific corpus. How does it work? ...
3
votes
1answer
5k views

Adding BERT embeddings in LSTM embedding layer

I am planning to use BERT embeddings in the LSTM embedding layer instead of the usual Word2vec/Glove Embeddings. What are the possible ways to do that?
4
votes
1answer
113 views

Will BERT embedding be always same for a given document when used as a feature extractor

When we use BERT embeddings for a classification task, would we get different embeddings every time we pass the same text through the BERT architecture? If yes, is it the right way to use the ...
1
vote
0answers
98 views

Multiple embedding layers?

How would one go about inputting multiple high dimensionality categorical columns using TensorFlow's Embedding Feature Columns? Does that even make sense to do? For example: for a car price predictor,...
1
vote
0answers
20 views

Creating a zero element in embedding space

I have some variable length input vectors for my own use case of a 'stylistic transfer'-esque process, and I am wondering if anyone knows of a way to engineer an input that maps to a 0 element in ...
2
votes
2answers
65 views

Do we have cross-language vector space for word embedding?

Do we have cross-language vector space for word embedding? When measure similarity for apple/Pomme/mela/Lacus/苹果/りんご, they should be the same If would be great if there's available internet service ...