What are the major layers in a Vision Transformer?

Question

Currently, I am studying deepfake detection using deep learning methods. Convolution neural networks, recurrent neural networks, long-short term memory networks, and vision transformers are famous deep learning-based methods that are used in deepfake detection, as I found in my study.

I was able to find that CNNs, RNNs and LSTMs are multilayered neural networks, but I found very little about the neural network layers in a Vision Transformer. (Like a typical CNN has an input layer, pooling layer, and a fully connected layer, and finally an output layer. RNN has an input layer, multiple hidden layers and an output layer.)

So, what are the main neural network layers in a Vision Transformer?

Answer 1

The Transformer family of architectures is a separate family of NN architectures, different from the CNNs and RNNs.

The main part of the Vision Transformer are the self-attention layers.

The architecture proposed in the paper An Image is Worth 16x16 Words treats each 16x16 as a word in the sentence. There is a convolutional layer (with kernel_size=16 and stride 16) that transforms the input patches into tokens as in NLP problem, and then these tokens are propagated through multiple layers.

Each Transformer encoder is a standard Transformer block, consisting of the:

• Multihead self-attention layer that transforms tokens into keys, queries and values
• Feedforward layer acting on each token independently (pointwise nonlinearity)
• LayerNormalization modules between them.

Each image is treated as a sentence or chunk of text. The main idea and advantage of self-attention layers is the ability to collect the global context of the given data sample, whereas CNNs are restricted to a neighboorhood of the given pixel (and can have a global understanding of the data after a sufficient number of convolutional layers).

If you are inexperienced with transformers, I recommend reading this blog as an easily accessible and comprehensive introduction to Transformers.