For an upcoming project, I am trying to build a neural network for classifying text from scratch, without the use of libraries. This requires an embedding layer, or a way to convert words to some vector representation. I understand the gist, but I can't find any deep explanations or tutorials that don't start with importing TensorFlow. All I'm really told is that it works by context using a few surrounding words, but I don't understand exactly how.

Is it much different from a classic network, with weights and biases? How does it figure out the loss?

If someone could point me towards a guide to how these things work exactly I would be very grateful.


1 Answer 1


Word2vec embeddings are trained using a simple auto-encoder model that takes a word and tries to predict one word from the window of surrounding words.

enter image description here

You could define it like this:

num_of_words = 50000
# one hot encoded word
input = Input(num_of_words)
# You could use non linear activation
w2v = Dense(300, activation=”linear”)(input)
output = Dense(num_of_words, activation=”softmax”)(w2v)

But in practice, the model is redefined and takes two words as input and predicts the next words. It outputs a probability score for all the words it knows (the model’s “vocabulary”, which can range from a few thousand to over a million words).

enter image description here

It is trained both ways from the beginning to the end of a sentence and in the reverse direction. The loss used is categorical_crossentropy. A detailed explanation can be found here [http://jalammar.github.io/illustrated-word2vec/]

  • $\begingroup$ thank you for this, ill read the article you have mentioned, but if I could also ask how many words is a good amount to use to train such a network? $\endgroup$ Commented Jul 12, 2020 at 13:13
  • 2
    $\begingroup$ It depends what you want to achieve And what kind of hardware you have. I trained it for 1 million words on a regular laptop with gpu. I did not want to reinvent the wheel so I have used gensim. If There is a pretrained model for your task/language then I would go for it as it usually is trained on bigger datasets. $\endgroup$
    – user301845
    Commented Jul 12, 2020 at 13:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .