Questions tagged [word2vec]

This tag is for questions relating to the language and models that create them.

Filter by
Sorted by
Tagged with
1
vote
1answer
38 views

Essay Grading LSTM not working

I have been making an essay Grading LSTM for the past week as a little project. The data used is a private dataset similar to ASAP essay grading dataset. I have get the network to work before but when ...
0
votes
0answers
9 views

Why word embedding such as word2vec is not used as the output layer of a seq2seq decoder?

It would make sense to make the decoder predict a smaller embedding vector instead of softmax over a large dictionary. The word having the most cosine similarity with the output embedding could be ...
0
votes
0answers
14 views

Confusion on how skip gram implementation is formulated

I'm using this source to understand the skip gram model. Let's say the context size is $4$ ($2$ context words on each side of the target word). This image illustrates how training examples are ...
2
votes
1answer
53 views

How does Continuous Bag of Words ensure that similar words are encoded as similar embeddings?

This is related to my earlier question, which I'm trying to break down into parts (this being the first). I'm reading notes on word vectors here. Specifically, I'm referring to section 4.2 on page 7. ...
0
votes
0answers
19 views

Understanding how continuous bag of words method learns embedded representations

I'm reading notes on word vectors here. Specifically, I'm referring to section 4.2 on page 7. First, regarding points 1 to 6 - here's my understanding: If we have a vocabulary $V$, the naive way to ...
1
vote
1answer
45 views

How do I classify strings with possibly no meaning?

I am quite new to text classification. Using EAST text detection model, I get multiple strings that aren't words and most often have no meaning. For example, IDs, brand names, etc. I would like to ...
0
votes
0answers
12 views

Embedding Gensim fast-text

Would you suggest to train my own Fast-text embedding using the Gensim library despite i have 1800 sentences and 2k vocabulary length? Don't you think there are too few words? or is there not a ...
2
votes
2answers
262 views

How to implement word2vec using Tensorflow 2.0 keras API?

Since Keras API as defined as layers, how would it be used to implement the word2vec?
3
votes
0answers
20 views

Why does all of NLP literature use noise contrastive estimation loss for negative sampling instead of sampled softmax loss?

A sampled softmax function is like a regular softmax but randomly selects a given number of 'negative' samples. This is difference than NCE Loss, which doesn't use a softmax at all, it uses a ...
1
vote
1answer
48 views

Skip-Gram Model Training

Suppose we want to predict context words $w_{i-h}, \dots, w_{i+h}$ given a target word $w_i$ for a window size $h$ around the target word $w_i$. We can represent this as: $$p(w_{i-h}, \dots, w_{i+h}|...
2
votes
0answers
39 views

How to perform unsupervised anomaly detection from log file with mostly textual data?

I have a log file of the format, Index, Date, Timestamp, Module, App, Context, Session, Verbosity level, Description The log file can be considered as a master log, which consists of individual ...
0
votes
1answer
53 views

What do the vectors of the center and outside word look like in word2vec?

In word2vec, the task is to learn to predict which words are most likely to be near each other in some long corpus of text. For each word $c$ in the corpus, the model outputs the probability ...
1
vote
1answer
51 views

Learning similarities between customers and offers representation

I am interested in a framework for learning the similarity of different input representations based on some common context. I have looked into word2vec, SVD and siamese networks, all of which are ...
1
vote
0answers
29 views

Does it make sense to add word embeddings as additional features for LSTM model?

I have an LSTM model. This model takes as input tokens. Those tokens represent XML markups extracted from some XML files. My model is working fine. However, I want to optimize it by adding word ...
3
votes
2answers
105 views

How should the output layer of an LSTM be when the output are word embeddings?

I'm having trouble grasping how to output word embeddings from an LSTM model. I'm seeing many examples using a softmax activation function on the output, but for that I would need to output one hot ...
4
votes
1answer
44 views

Do individual dimensions in vector space have meaning?

Word2vec assigns an N-dimensional vector to given words (which can be considered a form of dimensionality reduction). It turns out that, at least with a number of canonical examples, vector ...
-2
votes
1answer
107 views

I need a word database… Any qualities I should look for?

I need a word database to train from. I found a word2vec JS word vector database, but I need a method to teach it which words go in which patterns. Please note that I am not asking to have you ...