Questions tagged [vision-transformer]
The vision-transformer tag has no usage guidance.
19
questions
7
votes
2
answers
3k
views
Why class embedding token is added to the Visual Transformer?
In the famous work on the Visual Transformers, the image is split into patches of a certain size (say 16x16), and these patches are treated as tokens in the NLP tasks. In order to perform ...
- 2,564
4
votes
1
answer
2k
views
Do Vision Transformers handle arbitrary sequence lengths the same way as normal Transformers?
Does ViT do handle arbitrary sequence lengths using masking the same way the normal Transformer does?
The ViT paper doesn't mention anything about it, so I assume it uses masking like the normal ...
- 43
4
votes
1
answer
1k
views
How does the embeddings work in vision transformer from paper?
I get the part from the paper where the image is split into P say 16x16 (smaller images) patches and then you have to ...
- 253
3
votes
2
answers
359
views
Does the position of the tokens in Vision Transformer matter?
I am reading through the Vision Transformer paper and other related papers, such as DeiT and Visual Prompt Tuning (VPT). I wonder if the position of the tokens that flow through the Transformer encode ...
- 1,140
2
votes
1
answer
552
views
What are the major layers in a Vision Transformer?
Currently, I am studying deepfake detection using deep learning methods. Convolution neural networks, recurrent neural networks, long-short term memory networks, and vision transformers are famous ...
2
votes
2
answers
1k
views
Why does CLIP use a decoder-only transformer for encoding text?
In CLIP [1], the authors train a model to learn multi-modal (text, audio) embeddings by maximizing the cosine similarity between text and image embeddings produced by text and image encoders.
For the ...
- 309
2
votes
0
answers
60
views
Is it possible for original Vision Transformer (ViT) to do fine-grained semanantic segmentation? if so, how?
As far as I know, in the original ViT, the image is first divided to a fixed size of patch (16x16, for example) then they are flattened and treated as tokens and fed into Transformer.
Without using ...
- 21
2
votes
0
answers
49
views
What are the specific differences between vision transformers variants?
I have tried 4 different types of attacks on vision transformers (ViT small and tiny, DeiT small and tiny) but the attack successes on smaller versions are higher than the tiny versions. My ...
- 31
1
vote
1
answer
762
views
Which situation will helpful using encoder or decoder or both in transformer model?
I have some questions about using (encoder / decoder / encoder-decoder) transformer models, included (language) transformer or Vision transformer.
The overall form of a transformer consists of an ...
- 27
1
vote
1
answer
634
views
What is the difference between Mean Teacher and Knowledge Distillation?
I recently read two papers:
BYOL Bootstrap your own latent: A new approach to self-supervised Learning
DINO Emerging Properties in Self-Supervised Vision Transformers.
I am confused about the terms ...
0
votes
1
answer
55
views
What do we mean by the notation $\mathbf{x}_{p} \in \mathbb{R}^{N \times\left(P^{2} \cdot C\right)}$?
I was going through this VIT paper, what will it look like in torch , if we are trying to write this expression.
- 101
0
votes
1
answer
198
views
What is the difference between a vision transformer and image-based relational learning?
I am trying to figure out the difference between the architecture used in this and this paper. It looks like both used multi-headed self-attention and therefore should be the same in principle.
- 618
0
votes
1
answer
131
views
Can Vision Transformers be used to extract features?
Can Vision Transformers be used to extract features, just like with VGG ?
I am interested in using this vision transformer in extracting features (https://huggingface.co/google/vit-base-patch16-224)
...
0
votes
1
answer
2k
views
How "Patch Merging" works in SWIN-Transformers?
In the SOTA paper: SWIN-Transformers, the authors have tried their best to explain everything clearly. I have got an idea of how it works except the Patch Merging ...
- 253
0
votes
0
answers
41
views
ViT fails to detect a white pixel in a black image
I wanted to report you to some experiments in the context of Deep Learning for Computer Vision, in particular for visual reasoning.
The main question I try to answer is the difference between the ...
0
votes
0
answers
23
views
What algorithms could I use if I want to increase the accuracy of matched keypoints in an image pair?
Let's say that I used a keypoint detector like SIFT or SuperPoint to detect keypoints in image 1 and 2. Afterwards, I used a keypoint matcher to match corresponding keypoints in this image pair. The ...
- 101
0
votes
0
answers
55
views
How do i approach creating a masked auto-encoder for feature extraction
I trained Masked Autoencoder-based models in order to use the encoder as a backbone for another task. Pretraining has been done in a Self-Supervised manner on image data. Now that it comes to my ...
- 1
0
votes
0
answers
131
views
How does Position embeddings work in Vision Transformer
I'm a bit confused how the position embedding in happened to each patch in the transformer. I thought Ideally we'd want each patch to have a value of (1, 2, 3, 4....) to describe the position of the ...
- 3
0
votes
0
answers
20
views
How many pretraining image is enough for Swin Transformer?
Here is the spec of experiment setup:
We have 3D micro CT image of the rats, and we want to perform pretraining on such data. The image is masked, so only the portion around the backbone is visible.
...
