For an upcoming project, I am trying to build a neural network for classifying text from scratch, without the use of libraries. This requires an embedding layer, or a way to convert words to some vector representation. I understand the gist, but I can't find any deep explanations or tutorials that don't start with importing TensorFlow. All I'm really told is that it works by context using a few surrounding words, but I don't understand exactly how.

Is it much different from a classic network, with weights and biases? How does it figure out the loss?

If someone could point me towards a guide to how these things work exactly I would be very grateful.


Word2vec embedding are trained by simple auto-encoder model that takes a word and tries to predict one word form the window of surrounding words.

enter image description here

You could define it like this:

num_of_words = 50000
# one hot encoded word
input = Input(num_of_words)
# You could use non linear activation
w2v = Dense(300, activation=”linear”)(input)
output = Dense(num_of_words, activation=”softmax”)(w2v)

But in practice, the model is redefined and takes two words as input and predicts next words. And it outputs a probability score for all the words it knows (the model’s “vocabulary”, which can range from a few thousand to over a million words).

enter image description here

It is trained both ways from beginning to end of sentence and reverse. Loss is a regular categorical_crossentropy Detailed explenation can be found here [http://jalammar.github.io/illustrated-word2vec/]

  • $\begingroup$ thank you for this, ill read the article you have mentioned, but if I could also ask how many words is a good amount to use to train such a network? $\endgroup$ – JumpJehovah Jul 12 '20 at 13:13
  • $\begingroup$ It depends what you want to achieve And what kind of hardware you have. I trained it for 1 million words on a regular laptop with gpu. I did not want to reinvent the wheel so I have used gensim. If There is a pretrained model for your task/language then I would go for it as it usually is trained on bigger datasets. $\endgroup$ – user301845 Jul 12 '20 at 13:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.