Popular tokenizers use a special symbol such as "Ġ" (BPE) or "▁" (SentencePiece) to represent space. What is the reasoning behind this?

I did try searching for the answer. I got two types of explanations, but they don't explain anything to me.

Some languages such as Japanese don't typically use space to separate words.

That's true, but tokenizing such a language won't make use of the special character as well. I don't understand how having a token as "Ġhorse" is any different from " horse" in this scenario.

Let's assume we want to tokenize the made-up laguage Spaceless English, which is English, but without whitespace. tokenize("I'mridingahorse") -> ["I'm", "riding", "ahorse"]. No need for spaces at all, let alone a special character.

In fact this seems like it would be in favor of using a plain space as opposed to a special symbol.

Some tokens have a space in them and this helps us differentiate between spaces in the original text and spaces in the tokens

For example "New York" might be a token.

This kinda makes sense, but I still don't quite get why we would ever need this distinction. How is "ĠNew York" different from " New York"? " New York" is still different from [" New", " Toronto"]. A tokenizer would employ some kind of a greedy algorithm to tokenize text... and even if it's not greedy, but the full NP-complete search, it would still tokenize "I'm going to New York" as ["I'm", " going", " to", " New York"], the same way as we would expect it to tokenize "extracurricular" to say ["extra", "curricul", "ar"] and not to ["ex", "trac", "u", "rr", "ic", "u", "lar"], and the same way as we would expect it to work on Spaceless English.

Note that I'm looking at the problem only through the lens of converting text to tokens and tokens to text. Perhaps the need for these character has to do with creating the token list based on the training corpus?



You must log in to answer this question.