# How can I create an embedding layer to convert words to a vector space from scratch?

For an upcoming project, I am trying to build a neural network for classifying text from scratch, without the use of libraries. This requires an embedding layer, or a way to convert words to some vector representation. I understand the gist, but I can't find any deep explanations or tutorials that don't start with importing TensorFlow. All I'm really told is that it works by context using a few surrounding words, but I don't understand exactly how.

Is it much different from a classic network, with weights and biases? How does it figure out the loss?

If someone could point me towards a guide to how these things work exactly I would be very grateful.

## 1 Answer

Word2vec embedding are trained by simple auto-encoder model that takes a word and tries to predict one word form the window of surrounding words.

You could define it like this:

num_of_words = 50000
# one hot encoded word
input = Input(num_of_words)
# You could use non linear activation
w2v = Dense(300, activation=”linear”)(input)
output = Dense(num_of_words, activation=”softmax”)(w2v)


But in practice, the model is redefined and takes two words as input and predicts next words. And it outputs a probability score for all the words it knows (the model’s “vocabulary”, which can range from a few thousand to over a million words).

It is trained both ways from beginning to end of sentence and reverse. Loss is a regular categorical_crossentropy Detailed explenation can be found here [http://jalammar.github.io/illustrated-word2vec/]

• thank you for this, ill read the article you have mentioned, but if I could also ask how many words is a good amount to use to train such a network? – JumpJehovah Jul 12 at 13:13
• It depends what you want to achieve And what kind of hardware you have. I trained it for 1 million words on a regular laptop with gpu. I did not want to reinvent the wheel so I have used gensim. If There is a pretrained model for your task/language then I would go for it as it usually is trained on bigger datasets. – user301845 Jul 12 at 13:50