I wanted to report you to some experiments in the context of Deep Learning for Computer Vision, in particular for visual reasoning. The main question I am trying to answer is the difference between the Vision Transformer (ViT: https://arxiv.org/abs/2010.11929v2) and CNNs in visual reasoning benchmarks that require object localization.


Given an NxN black image with a single white pixel, predict the coordinates of the white pixel. The following are some properties of the dataset:

the cardinality of the dataset is |NxN|, i.e. the number of all possible black images of size NxN with one white pixel in the experiments carried out, the training set and the validation set have a proportion of 80%/20%, split with a uniform distribution

I attach the image of a sample entitled the ground truth, i.e. the coordinates of the white pixel. The shared (between the two tested models) hyperparameters used are:

  • Optimizer: SGD
  • Momentum: 0.9
  • LR: 7e-4
  • Batch Size: 8 for ViT, 256 for CNNs

ViT hyperparameters used are:

  • num_layers: 4
  • num_head: 2
  • dropout: 0
  • patch_size: 1
  • embedding_dim: 8

(both learned and fixed positional encoding are tested)

The CNN is a ResNet (without stem layer and max_pooling) with 8 layers (stride = 2 for layers 1, 3, 5, 7 and channels doubled), the channel sequence for each layer is: 1 (input)-3-3-8-8-16-16-32-32.

The output of the two models is obviously the same, the pair of coordinates of the white pixel, for this reason, both L1 and MSE loss were tested (the results do not change).

Finally, CoordConv encoding was also tested (https://arxiv.org/abs/1807.03247). It consists of creating two channels to be concatenated to the input image which represents the coordinates of the pixels.


The goal of these experiments is the evaluation of the Vision Transformer and CNNs on a visual reasoning task that requires localization. In particular, I expect the choice of patch size for the Vision Transformer to be critical to convergence. In particular, I expected that by choosing a patch size = 1 the ViT would be able to overfit the training set (thanks to positional encoding).


the results showed that:

  • The ViT transformer works at chance level for any patch size, it is not able to overfit the training set. In particular, it returns the same output for any input, behaving shift-invariant (permutation invariant in the pacification view).
  • ResNet is able to overfit the training set with a good level of generalization, but by increasing N (i.e. the size of the input images and therefore the cardinality of the representation space) the results worsen
  • CoordResnet on the other hand is able to overfit and generalize well for any N

How do you explain the behavior of the Vision Transformer? enter image description here



You must log in to answer this question.