Skip to main content

Questions tagged [vision-transformer]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
8 votes
2 answers
5k views

Why class embedding token is added to the Visual Transformer?

In the famous work on the Visual Transformers, the image is split into patches of a certain size (say 16x16), and these patches are treated as tokens in the NLP tasks. In order to perform ...
spiridon_the_sun_rotator's user avatar
4 votes
1 answer
2k views

Which situation will helpful using encoder or decoder or both in transformer model?

I have some questions about using (encoder / decoder / encoder-decoder) transformer models, included (language) transformer or Vision transformer. The overall form of a transformer consists of an ...
Yang's user avatar
  • 57
4 votes
1 answer
2k views

Do Vision Transformers handle arbitrary sequence lengths the same way as normal Transformers?

Does ViT do handle arbitrary sequence lengths using masking the same way the normal Transformer does? The ViT paper doesn't mention anything about it, so I assume it uses masking like the normal ...
Dean R's user avatar
  • 43
4 votes
1 answer
2k views

How does the embeddings work in vision transformer from paper?

I get the part from the paper where the image is split into P say 16x16 (smaller images) patches and then you have to ...
Deshwal's user avatar
  • 263
3 votes
2 answers
535 views

Does the position of the tokens in Vision Transformer matter?

I am reading through the Vision Transformer paper and other related papers, such as DeiT and Visual Prompt Tuning (VPT). I wonder if the position of the tokens that flow through the Transformer encode ...
Minh-Long Luu's user avatar
3 votes
2 answers
2k views

Why does CLIP use a decoder-only transformer for encoding text?

In CLIP [1], the authors train a model to learn multi-modal (text, vision) embeddings by maximizing the cosine similarity between text and image embeddings produced by text and image encoders. For the ...
thesofakillers's user avatar
2 votes
1 answer
613 views

What are the major layers in a Vision Transformer?

Currently, I am studying deepfake detection using deep learning methods. Convolution neural networks, recurrent neural networks, long-short term memory networks, and vision transformers are famous ...
Pawara Siriwardhane's user avatar
2 votes
0 answers
61 views

Is it possible for original Vision Transformer (ViT) to do fine-grained semanantic segmentation? if so, how?

As far as I know, in the original ViT, the image is first divided to a fixed size of patch (16x16, for example) then they are flattened and treated as tokens and fed into Transformer. Without using ...
wanburana's user avatar
2 votes
0 answers
54 views

What are the specific differences between vision transformers variants?

I have tried 4 different types of attacks on vision transformers (ViT small and tiny, DeiT small and tiny) but the attack successes on smaller versions are higher than the tiny versions. My ...
Craving_gold's user avatar
1 vote
1 answer
780 views

What is the difference between Mean Teacher and Knowledge Distillation?

I recently read two papers: BYOL Bootstrap your own latent: A new approach to self-supervised Learning DINO Emerging Properties in Self-Supervised Vision Transformers. I am confused about the terms ...
Đặng Huy Hoàng's user avatar
1 vote
0 answers
275 views

Glass Degradation Video Prediction

We are working on the subject above, where a sequence of $n$ glass frames forms an example with an associate target that is a video of the glass future state. We would like to know if there exists an ...
Filippo Portera's user avatar
0 votes
1 answer
59 views

What do we mean by the notation $\mathbf{x}_{p} \in \mathbb{R}^{N \times\left(P^{2} \cdot C\right)}$?

I was going through this VIT paper, what will it look like in torch , if we are trying to write this expression.
TheExorcist's user avatar
0 votes
1 answer
214 views

What is the difference between a vision transformer and image-based relational learning?

I am trying to figure out the difference between the architecture used in this and this paper. It looks like both used multi-headed self-attention and therefore should be the same in principle.
desert_ranger's user avatar
0 votes
1 answer
340 views

Can Vision Transformers be used to extract features?

Can Vision Transformers be used to extract features, just like with VGG ? I am interested in using this vision transformer in extracting features (https://huggingface.co/google/vit-base-patch16-224) ...
Ahmed Gamal's user avatar
0 votes
1 answer
2k views

How "Patch Merging" works in SWIN-Transformers?

In the SOTA paper: SWIN-Transformers, the authors have tried their best to explain everything clearly. I have got an idea of how it works except the Patch Merging ...
Deshwal's user avatar
  • 263
0 votes
0 answers
27 views

Where U-Net and Convolutional layers are settled in Stable Diffusion model?

When I read about Stable Diffusion model, they usually talk about adjusting convolution layers or U-Net weights. I believe they both should be related together and the U-Net is the part that accepts ...
best_of_man's user avatar
0 votes
0 answers
12 views

How to train ViT on smaller datasets?

I know ViTs aren't made for small datasets and low resolution. But have you ever reached traditional CNN accuracy using ViT on CIFAR10/100. I have been playing around with ViT on CIFAR10 and 100. But ...
v1998199904's user avatar
0 votes
0 answers
83 views

ViT fails to detect a white pixel in a black image

I wanted to report you to some experiments in the context of Deep Learning for Computer Vision, in particular for visual reasoning. The main question I am trying to answer is the difference between ...
Fiorenzo Parascandolo's user avatar
0 votes
0 answers
25 views

What algorithms could I use if I want to increase the accuracy of matched keypoints in an image pair?

Let's say that I used a keypoint detector like SIFT or SuperPoint to detect keypoints in image 1 and 2. Afterwards, I used a keypoint matcher to match corresponding keypoints in this image pair. The ...
user402016's user avatar
0 votes
0 answers
110 views

How do i approach creating a masked auto-encoder for feature extraction

I trained Masked Autoencoder-based models in order to use the encoder as a backbone for another task. Pretraining has been done in a Self-Supervised manner on image data. Now that it comes to my ...
Mitch's user avatar
  • 1
0 votes
0 answers
170 views

How does Position embeddings work in Vision Transformer

I'm a bit confused how the position embedding in happened to each patch in the transformer. I thought Ideally we'd want each patch to have a value of (1, 2, 3, 4....) to describe the position of the ...
a__ys's user avatar
  • 3