3
$\begingroup$

I am reading through the Vision Transformer paper and other related papers, such as DeiT and Visual Prompt Tuning (VPT). I wonder if the position of the tokens that flow through the Transformer encode matters? Let's break it down, assuming we use ViT 16x16:

  • The original Vision Transformer has 1 classifier [CLS] token, along with 16x16=196 image tokens. In general, we have 197 tokens overall, where the position is: [[CLS], [Image Patches] x 196]
  • DeiT further adds one Distillation token to the end of the sequence, so we have 198 overall: [[CLS], [Image Patches] x 196, [Distill Token]]
  • However, I notice a very strange thing with VPT that they add the Prompt token after the [CLS] but before the Image patches. So the position of the tokens in VPT is: [[CLS], [Prompts] x N, [Image Patches] x 196].

I wonder does such a thing matter? What if we change the position of these tokens, e.g., putting the Prompt tokens to the last?

$\endgroup$

2 Answers 2

3
+50
$\begingroup$

It should not matter. To explain why, we need to understand how a transformer works.

Transformers were originally designed for language models. They compute a self attention matrix, which is a fancy way of saying they input a sequence, and learn how every part of that sequence relates to every other part of the sequence.

Since self attention is computed for every token with respect to every other token, transformers are completely agnostic to the order in which tokens are placed.

For some applications this is a desirable property, but for others, we want to give the transformer some knowledge of the sequence order. This is done by adding positional embeddings. These are added to the sequence prior to being passed into the transformer, and they are used to encode sequence information.

So to review:

  1. Vanilla transformers are totally agnostic to the sequence order.
  2. For applications where this is undesirable a positional encoding is added.
  3. The changing order that you observed should not matter.

This blog post may be helpful for further information.

$\endgroup$
2
$\begingroup$

It does not matter.

Although, I can imagine a situation where it could matter a bit - when position embeddings are not learnt but calculated and fixed like in the original transformer (Attention is all you need). Then the attention mechanism may be slightly biased towards one or another position. But AFAIR in ViT the position embeddings are learnt. Not sure about other papers you mention.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .