What are the approaches of encoding text data? I would be glad to hear some summarization from experienced persons.
And are there any solutions accepting words outside the vocabulary and including them to the results (online machine learning)?
Data input
So my basic understanding is that if we want predict some value (linear regression) or say what is the probability of occuring some event (logistic regression) we have to gather some features as our input and encode them as number. But this is not necessarily true when working with continuous data like sentences.
The most naive aproach, which comes to my mind is just to assign some natural numbers to each word in the vocabulary. But this number does not contain any meaningful data about the word itself. On the other hand what seems to be important in NLP is just the order of the words. This is where I think about n-grams so we feed network with more than just one word. Or attention like in the Transformer.
Another idea, which cames to my mind is to vectorize the word using one of the Word Embedding technique. Here we have some context about the word so the input is not just a dumb number. But does it have any value when we want to predict the next word? Can Word Embedding be used in that way or it's purpose is completely different.
Last thing I was reading of was to encode characters rather than words but it feels pointless in such basic example as next word prediction. I would think about it more for sub-word tasks like inflection generating.
Labelling
Again based on my knowledge when we want to solve yes/no problem we're using sigmoid function. If we have more classes we can use one-hot encoding. But sometimes the output of the network might give us ambiguous meaning so we're using the softmax function so all output sum to 1.
How this looks in NLP area? When having a vocabulary consisting of 600k words do we really need 600k softmaxed outputs? I'm also thinking there about Word Embedding solutions where we can reduce the number of outputs to let's say 300 numbers and then find the closest word matching the output without using softmax.