Questions tagged [vision-transformer]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
23 views

What algorithms could I use if I want to increase the accuracy of matched keypoints in an image pair?

Let's say that I used a keypoint detector like SIFT or SuperPoint to detect keypoints in image 1 and 2. Afterwards, I used a keypoint matcher to match corresponding keypoints in this image pair. The ...
user402016's user avatar
0 votes
1 answer
117 views

Can Vision Transformers be used to extract features?

Can Vision Transformers be used to extract features, just like with VGG ? I am interested in using this vision transformer in extracting features (https://huggingface.co/google/vit-base-patch16-224) ...
Ahmed Gamal's user avatar
1 vote
1 answer
698 views

Which situation will helpful using encoder or decoder or both in transformer model?

I have some questions about using (encoder / decoder / encoder-decoder) transformer models, included (language) transformer or Vision transformer. The overall form of a transformer consists of an ...
Yang's user avatar
  • 27
0 votes
0 answers
55 views

How do i approach creating a masked auto-encoder for feature extraction

I trained Masked Autoencoder-based models in order to use the encoder as a backbone for another task. Pretraining has been done in a Self-Supervised manner on image data. Now that it comes to my ...
Mitch's user avatar
  • 1
2 votes
0 answers
60 views

Is it possible for original Vision Transformer (ViT) to do fine-grained semanantic segmentation? if so, how?

As far as I know, in the original ViT, the image is first divided to a fixed size of patch (16x16, for example) then they are flattened and treated as tokens and fed into Transformer. Without using ...
wanburana's user avatar
0 votes
0 answers
122 views

How does Position embeddings work in Vision Transformer

I'm a bit confused how the position embedding in happened to each patch in the transformer. I thought Ideally we'd want each patch to have a value of (1, 2, 3, 4....) to describe the position of the ...
a__ys's user avatar
  • 3
0 votes
0 answers
20 views

How many pretraining image is enough for Swin Transformer?

Here is the spec of experiment setup: We have 3D micro CT image of the rats, and we want to perform pretraining on such data. The image is masked, so only the portion around the backbone is visible. ...
Sherry Yuan's user avatar
2 votes
2 answers
1k views

Why does CLIP use a decoder-only transformer for encoding text?

In CLIP [1], the authors train a model to learn multi-modal (text, audio) embeddings by maximizing the cosine similarity between text and image embeddings produced by text and image encoders. For the ...
thesofakillers's user avatar
3 votes
2 answers
346 views

Does the position of the tokens in Vision Transformer matter?

I am reading through the Vision Transformer paper and other related papers, such as DeiT and Visual Prompt Tuning (VPT). I wonder if the position of the tokens that flow through the Transformer encode ...
Minh-Long Luu's user avatar
0 votes
1 answer
55 views

What do we mean by the notation $\mathbf{x}_{p} \in \mathbb{R}^{N \times\left(P^{2} \cdot C\right)}$?

I was going through this VIT paper, what will it look like in torch , if we are trying to write this expression.
TheExorcist's user avatar
2 votes
0 answers
49 views

What are the specific differences between vision transformers variants?

I have tried 4 different types of attacks on vision transformers (ViT small and tiny, DeiT small and tiny) but the attack successes on smaller versions are higher than the tiny versions. My ...
Craving_gold's user avatar
0 votes
1 answer
2k views

How "Patch Merging" works in SWIN-Transformers?

In the SOTA paper: SWIN-Transformers, the authors have tried their best to explain everything clearly. I have got an idea of how it works except the Patch Merging ...
Deshwal's user avatar
  • 253
1 vote
1 answer
631 views

What is the difference between Mean Teacher and Knowledge Distillation?

I recently read two papers: BYOL Bootstrap your own latent: A new approach to self-supervised Learning DINO Emerging Properties in Self-Supervised Vision Transformers. I am confused about the terms ...
Đặng Huy Hoàng's user avatar
4 votes
1 answer
2k views

Do Vision Transformers handle arbitrary sequence lengths the same way as normal Transformers?

Does ViT do handle arbitrary sequence lengths using masking the same way the normal Transformer does? The ViT paper doesn't mention anything about it, so I assume it uses masking like the normal ...
Dean R's user avatar
  • 43
2 votes
1 answer
544 views

What are the major layers in a Vision Transformer?

Currently, I am studying deepfake detection using deep learning methods. Convolution neural networks, recurrent neural networks, long-short term memory networks, and vision transformers are famous ...
Pawara Siriwardhane's user avatar
0 votes
1 answer
197 views

What is the difference between a vision transformer and image-based relational learning?

I am trying to figure out the difference between the architecture used in this and this paper. It looks like both used multi-headed self-attention and therefore should be the same in principle.
desert_ranger's user avatar
7 votes
2 answers
3k views

Why class embedding token is added to the Visual Transformer?

In the famous work on the Visual Transformers, the image is split into patches of a certain size (say 16x16), and these patches are treated as tokens in the NLP tasks. In order to perform ...
spiridon_the_sun_rotator's user avatar
4 votes
1 answer
1k views

How does the embeddings work in vision transformer from paper?

I get the part from the paper where the image is split into P say 16x16 (smaller images) patches and then you have to ...
Deshwal's user avatar
  • 253