1
$\begingroup$

In pre-processing of text, we need to assign a number for each token in a text. Then only we can pass it to a model. In pre-processing of text, we need to assign a number for each token in a text. The paragraph from this section named Text Preprocessing recommended indexing according to the frequency of the token

The string type of the token is inconvenient to be used by models, which take numerical inputs. Now let us build a dictionary, often called vocabulary as well, to map string tokens into numerical indices starting from 0. To do so, we first count the unique tokens in all the documents from the training set, namely a corpus, and then assign a numerical index to each unique token according to its frequency. Rarely appeared tokens are often removed to reduce the complexity. Any token that does not exist in the corpus or has been removed is mapped into a special unknown token “”. We optionally add a list of reserved tokens, such as “” for padding, “” to present the beginning for a sequence, and “” for the end of a sequence.

I want to know whether it is necessary to index in accordance with the frequency of token or any unique index serves the purpose?

$\endgroup$

2 Answers 2

2
$\begingroup$

The main reason is given in the next sentence, i.e. after crating a corpus we want to know which words are the most frequent and which one are the most rare.

All together, rare and frequent words are usually referred to as stop words, and they use to preprocess the data, specifically they are usually removed (with some exceptions).

Another reason to keep trace of the frequency of words is that to convert text to numbers all approaches start by computing the probability of a word occuring in the corpus, hence counting unique values and their frequency in one shot will save computational time in later steps.

$\endgroup$
2
$\begingroup$

In NLP, word types (not tokens) are often represented by numbers, as they are easier to process: They all take the same space, so random access is a lot easier than with variable length strings, and comparison is also a lot faster. The actual numerical value used is completely arbitrary.

If you have a static corpus you want to process (ie you know all the words that occur in it, and there won't ever be any new words coming in), then you can kill two birds with one stone and double up the numerical value with their frequency rank (not the actual frequency, as there will be many words which have the same frequency).

I guess one reason with language models is that they often have a fixed number of words that they can handle. If we arbitrarily set that to 1024, we can then make sure that only the most frequent words are used, by simply replacing any index value greater than 1022 by 1023 (assuming a range from 0 to 1023). The word type with the index 1023 then becomes "UNKNOWN". This is an elegant way to make sure that only the least common word types are excluded.

In an open-ended NLP system, you cannot do that (as the vocabulary is not stationary), so you'd usually have maintain a separate frequency list. If you then want to extract the 1023 most common word types, you'd always have to add a frequency-rank lookup each time, which you can save if the index is already the frequency rank.

So it's an efficiency gain in a trade-off for requiring a static vocabulary.

Please note that these are not stop words, as mentioned in another answer: stop words are high-frequency words, usually function words, which have no lexical meaning (but provide the 'glue' between words in a sentence). In days when memory was limited, these words would often be excluded, as they did not contribute much to the 'meaning' of a text (for a very limited definition of meaning). In a "bag-of-words" approach, they can probably be ignored. Rare words are not stop words.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .