I've seen two approaches for introducing custom tokens for transfer learning with large language models like Bert or GPT3. Some approaches introduce new tokens into the vocabulary and learn embeddings from scratch. This is the "traditional" approach. However, I've seen other papers that imitate custom tokens with the use of punctuation, e.g. "<custom-token>". In this case the model is not learning any new tokens, but is learning to connect subword tokens and punctation already in its vocabulary. I think this approach is often used with GPT3, as the closed API prevents learning new tokens from-scratch.
Has any research benchmarked whether one approach is better than another, when both options are available?
