I am reading through the Vision Transformer paper and other related papers, such as DeiT and Visual Prompt Tuning (VPT). I wonder if the position of the tokens that flow through the Transformer encode matters? Let's break it down, assuming we use ViT 16x16:
- The original Vision Transformer has 1 classifier
[CLS]token, along with 16x16=196 image tokens. In general, we have 197 tokens overall, where the position is:
[[CLS], [Image Patches] x 196]
- DeiT further adds one Distillation token to the end of the sequence, so we have 198 overall:
[[CLS], [Image Patches] x 196, [Distill Token]]
- However, I notice a very strange thing with VPT that they add the Prompt token after the
[CLS]but before the Image patches. So the position of the tokens in VPT is:
[[CLS], [Prompts] x N, [Image Patches] x 196].
I wonder does such a thing matter? What if we change the position of these tokens, e.g., putting the Prompt tokens to the last?