I know ViTs aren't made for small datasets and low resolution. But have you ever reached traditional CNN accuracy using ViT on CIFAR10/100.

I have been playing around with ViT on CIFAR10 and 100. But am not able to get it over 75% accuracy on CIFAR10.

I have tried these configurations for the architecture:

patch sizes: 4 and 8
dimensions: 512 and 786
depth/transformer blocks: 8 and 10
attention heads: 8, 10 and 12
mlp dimension: 512, 2048 and 3072

I have tried SGD and Adam with different learning rates (0.1, 0.01) and (0.01, 0.001) respectively, with learning rate step decreasing after 100 and 175 epochs.

Also using weight decay of 1e-4 and 1e-5 with some image augmentation such as random flip, random rotate, minimal color jittering, and random affines.

Any suggestions to improve the validation accuracy to reach ~90% ?



You must log in to answer this question.