Currently, I am studying deepfake detection using deep learning methods. Convolution neural networks, recurrent neural networks, long-short term memory networks, and vision transformers are famous deep learning-based methods that are used in deepfake detection, as I found in my study.

I was able to find that CNNs, RNNs and LSTMs are multilayered neural networks, but I found very little about the neural network layers in a Vision Transformer. (Like a typical CNN has an input layer, pooling layer, and a fully connected layer, and finally an output layer. RNN has an input layer, multiple hidden layers and an output layer.)

So, what are the main neural network layers in a Vision Transformer?


1 Answer 1


The Transformer family of architectures is a separate family of NN architectures, different from the CNNs and RNNs.

The main part of the Vision Transformer are the self-attention layers.

The architecture proposed in the paper An Image is Worth 16x16 Words treats each 16x16 as a word in the sentence. There is a convolutional layer (with kernel_size=16 and stride 16) that transforms the input patches into tokens as in NLP problem, and then these tokens are propagated through multiple layers.

enter image description here

Each Transformer encoder is a standard Transformer block, consisting of the:

  • Multihead self-attention layer that transforms tokens into keys, queries and values
  • Feedforward layer acting on each token independently (pointwise nonlinearity)
  • LayerNormalization modules between them.

enter image description here

Each image is treated as a sentence or chunk of text. The main idea and advantage of self-attention layers is the ability to collect the global context of the given data sample, whereas CNNs are restricted to a neighboorhood of the given pixel (and can have a global understanding of the data after a sufficient number of convolutional layers).

If you are inexperienced with transformers, I recommend reading this blog as an easily accessible and comprehensive introduction to Transformers.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .