I have tried 4 different types of attacks on vision transformers (ViT small and tiny, DeiT small and tiny) but the attack successes on smaller versions are higher than the tiny versions. My understanding is that the smaller versions are stronger than the tiny ones. I expect the results to be higher on tiny than smaller versions.
I am really confused. Is there any explanation for this?
What are the specific differences between the vision transformer models apart from the number of parameters and the embedding dimension?
Edit: I am loading these models from timm
