General approaches in text encoding and labelling for NLP [closed]

Question

What are the approaches of encoding text data? I would be glad to hear some summarization from experienced persons.

And are there any solutions accepting words outside the vocabulary and including them to the results (online machine learning)?

Data input

So my basic understanding is that if we want predict some value (linear regression) or say what is the probability of occuring some event (logistic regression) we have to gather some features as our input and encode them as number. But this is not necessarily true when working with continuous data like sentences.

The most naive aproach, which comes to my mind is just to assign some natural numbers to each word in the vocabulary. But this number does not contain any meaningful data about the word itself. On the other hand what seems to be important in NLP is just the order of the words. This is where I think about n-grams so we feed network with more than just one word. Or attention like in the Transformer.

Another idea, which cames to my mind is to vectorize the word using one of the Word Embedding technique. Here we have some context about the word so the input is not just a dumb number. But does it have any value when we want to predict the next word? Can Word Embedding be used in that way or it's purpose is completely different.

Last thing I was reading of was to encode characters rather than words but it feels pointless in such basic example as next word prediction. I would think about it more for sub-word tasks like inflection generating.

Labelling

Again based on my knowledge when we want to solve yes/no problem we're using sigmoid function. If we have more classes we can use one-hot encoding. But sometimes the output of the network might give us ambiguous meaning so we're using the softmax function so all output sum to 1.

How this looks in NLP area? When having a vocabulary consisting of 600k words do we really need 600k softmaxed outputs? I'm also thinking there about Word Embedding solutions where we can reduce the number of outputs to let's say 300 numbers and then find the closest word matching the output without using softmax.

Answer 1

Text encoding depends very much on the purpose of your application. Here are some examples:

Text-to-speech: You would start with the word form itself, and probably look it up in a table that gives you a mapping to the phonological structure. Or you work through it (eg with a finite state transducer) and look at combinations of letters and check for the most common realistion of these letters (depending on the language and the writing system you are using). In the latter approach you can also deal with unknown words.

Part-of-speech tagging: You would look up the string in a dictionary, returning you a list of all the possible tags for the word (possibly augmented by frequency information). From then on you would operate on a sequence of tags, until you have them disambiguated, and you end up with a list that maps each word in the sentence (token) with a particular tag. For unknown words you can either assign them all possible tags (and let the disambiguation sort them out), or choose only the open class ones (you won't find many unknown prepositions or determiners).

Information retrieval/storage: This is one situation where you would want to replace word by numbers. You'd have a look-up table that maps word to integer values, and these can be very efficiently stored. For example, in the Bank of English one of the biggest early language corpora (several 100s of millions of tokens in the early 1990s), each token was encoded as a 4 byte integer. It was then very fast to go from the index positions of a word (from an inveretd index) to the actual position in the file, as each token is 4 bytes long, so the token at index $n$ is at file position $4n$. It also somewhat compresses the text if you assume the average word length is greater than 4 (and that you don't encode the spaces between the tokens). Unknown words: they are by definition not in your text, so you can just return "not found".

Machine learning: ML approaches don't really work on strings, so you need to encode word numerically. Ideally this should not be a random mapping (such as alphabetical order), so one approach that is used is to represent each word by a vector that encodes which other words occur in its neighbourhood. It is a well-known principle in corpus linguistics (since the 1930s) that words are not randomly distributed in a text, but that some words co-occur with regularity. For example, rasher and bacon usually go together. So the vector which describes rasher will have a positive number in the slot which relates to does it occur with 'bacon'?, whereas the vector describing bookshelf will not. With a large enough vector, you can approximate similarities in meaning between words (based on their lexcial environments). Unknown words: you could use a random or average vector -- I don't know what the standard way is to deal with that.