# General approaches in text encoding and labelling for NLP [closed]

What are the approaches of encoding text data? I would be glad to hear some summarization from experienced persons.

And are there any solutions accepting words outside the vocabulary and including them to the results (online machine learning)?

Data input

So my basic understanding is that if we want predict some value (linear regression) or say what is the probability of occuring some event (logistic regression) we have to gather some features as our input and encode them as number. But this is not necessarily true when working with continuous data like sentences.

The most naive aproach, which comes to my mind is just to assign some natural numbers to each word in the vocabulary. But this number does not contain any meaningful data about the word itself. On the other hand what seems to be important in NLP is just the order of the words. This is where I think about n-grams so we feed network with more than just one word. Or attention like in the Transformer.

Another idea, which cames to my mind is to vectorize the word using one of the Word Embedding technique. Here we have some context about the word so the input is not just a dumb number. But does it have any value when we want to predict the next word? Can Word Embedding be used in that way or it's purpose is completely different.

Last thing I was reading of was to encode characters rather than words but it feels pointless in such basic example as next word prediction. I would think about it more for sub-word tasks like inflection generating.

Labelling

Again based on my knowledge when we want to solve yes/no problem we're using sigmoid function. If we have more classes we can use one-hot encoding. But sometimes the output of the network might give us ambiguous meaning so we're using the softmax function so all output sum to 1.

How this looks in NLP area? When having a vocabulary consisting of 600k words do we really need 600k softmaxed outputs? I'm also thinking there about Word Embedding solutions where we can reduce the number of outputs to let's say 300 numbers and then find the closest word matching the output without using softmax.

• What do you want to do? NLP is a vast area, and there are lots of options to deal with it, but this depends on what you want to achieve. Without knowing that it's impossible to answer. Nov 22 '21 at 9:38
• I wasn't thinking about any specific use case. I thought it will be possible to get some general understanding of that topic. But if I had to make something then I have two examples: 1. Conversion of text to phonetic representation in a language having some general rules but also many exceptions 2. Part-of-speech tagging Nov 22 '21 at 10:13
• Hello. Welcome to Artificial Intelligence Stack Exchange. After having quickly read several parts of this post, it seems to me that you're asking many distinct (although related) questions. Please, ask only one question per post. If you have more than one question/issue/problem, ask each question in a separate post. If you think this question was wrongly closed and you're actually asking the same question but in different ways, let me know.
– nbro
Nov 22 '21 at 11:46

Information retrieval/storage: This is one situation where you would want to replace word by numbers. You'd have a look-up table that maps word to integer values, and these can be very efficiently stored. For example, in the Bank of English one of the biggest early language corpora (several 100s of millions of tokens in the early 1990s), each token was encoded as a 4 byte integer. It was then very fast to go from the index positions of a word (from an inveretd index) to the actual position in the file, as each token is 4 bytes long, so the token at index $$n$$ is at file position $$4n$$. It also somewhat compresses the text if you assume the average word length is greater than 4 (and that you don't encode the spaces between the tokens). Unknown words: they are by definition not in your text, so you can just return "not found".