For LLM how is the token dictionary built?

Question

For LLM, how was decided how big and how many unique tokens exist for the english language?

Answer 1

For LLM, how was decided how big and how many unique tokens exist for the English language?

Empirically. E.g., see some info from Tokenizer Choice For LLM Training: Negligible or Crucial?:

Analyzing the impact of the vocabulary size revealed that in the monolingual English setting, the smaller/medium-sized, i.e., a vocabulary size of 33k/50k performs better (Table 5) whereas in the multilingual setting, in all cases except for German, larger vocabulary sizes result in better downstream performance. Taking into account the results presented in Table 3 showing that in the monolingual English setting, the best-performing tokenizer on average across all tasks had a vocabulary size of 33k and that the best-performing multilingual tokenizer had a vocabulary size of 100k additionally supports the observation that for the monolingual English setting a small vocabulary size is beneficial and for the multilingual setting a large vocabulary size is required.

Our findings in Fig. 3 show that increasing the vocabulary size from 50k to larger vocabulary sizes increases the computational costs in all cases.