In pre-processing of text, we need to assign a number for each token in a text. Then only we can pass it to a model. In pre-processing of text, we need to assign a number for each token in a text. The paragraph from this section named Text Preprocessing recommended indexing according to the frequency of the token
The string type of the token is inconvenient to be used by models, which take numerical inputs. Now let us build a dictionary, often called vocabulary as well, to map string tokens into numerical indices starting from 0. To do so, we first count the unique tokens in all the documents from the training set, namely a corpus, and then assign a numerical index to each unique token according to its frequency. Rarely appeared tokens are often removed to reduce the complexity. Any token that does not exist in the corpus or has been removed is mapped into a special unknown token “”. We optionally add a list of reserved tokens, such as “” for padding, “” to present the beginning for a sequence, and “” for the end of a sequence.
I want to know whether it is necessary to index in accordance with the frequency of token or any unique index serves the purpose?