As far as I know, in the original ViT, the image is first divided to a fixed size of patch (16x16, for example) then they are flattened and treated as tokens and fed into Transformer.
Without using later more recent techniques (such as Hierarchical patch merging in Swin transformer), I feel like it is not possible to do the fine-grained segmentation at all, the best it can do is to label the whole token (with 16x16 pixel) as one class, since the model no longer understands any spatial information in the token.
Is my understanding correct? I know that it didnt do as well as Swin Transformer which use more fine-grained hierachical patch merging. But can ViT still selectively label some pixels in one patch as one class and the other pixels as another class?
