In the famous work on the Visual Transformers, the image is split into patches of a certain size (say 16x16), and these patches are treated as tokens in the NLP tasks. In order to perform classification, a CLS token is added at the beginning of the resulting sequence: $$ [\textbf{x}_{class}, \textbf{x}_{p}^{1}, \ldots, \textbf{x}_{p}^{N}] ,$$ where $ \textbf{x}_{p}^{i}$ are image patches. There multiple layers in the architecture and the state of the CLS token on the output layer is used for classification.

I think this architectural solution is done in the spirit of NLP problems (BERT in particular). However, for me, it would be more natural not to create a special token, but perform 1d Global Pooling in the end, and attach an nn.Linear(embedding_dim, num_classes) as more conventional CV approach.

Why it is not done in this way? Or is there some intuition or evidence that this would perform worse than the approach used in the paper?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.