Skip to main content
8 votes
Accepted

Which situation will helpful using encoder or decoder or both in transformer model?

The original transformer paper presents the transformer as a model consisting of both encoder and decoder. However, many times you will see (or hear) people describing their model as a "...
pi-tau's user avatar
  • 815
5 votes
Accepted

Do Vision Transformers handle arbitrary sequence lengths the same way as normal Transformers?

Yes, they can handle sequences with arbitrary length sequence, but with some remarks. In the paper Training data-efficient image transformers & distillation through attention authors train models ...
spiridon_the_sun_rotator's user avatar
4 votes
Accepted

What are the major layers in a Vision Transformer?

The Transformer family of architectures is a separate family of NN architectures, different from the CNNs and RNNs. The main part of the Vision Transformer are the self-attention layers. The ...
spiridon_the_sun_rotator's user avatar
3 votes
Accepted

Does the position of the tokens in Vision Transformer matter?

It should not matter. To explain why, we need to understand how a transformer works. Transformers were originally designed for language models. They compute a self attention matrix, which is a fancy ...
chessprogrammer's user avatar
3 votes
Accepted

Why class embedding token is added to the Visual Transformer?

In their official repository, the author confirmed in the issue that the cls is not really important for the ViT, but they wanted to keep the ViT to be exactly the same with the NLP-Transformer, so ...
CuCaRot's user avatar
  • 912
2 votes
Accepted

What is the difference between a vision transformer and image-based relational learning?

An Image is Worth 16X16 Words: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE A Transformer consists of alternating layers of multiheaded self-attention. The Transformer Paper adapts a NLP architecture ...
Andre Goulart's user avatar
2 votes

How does the embeddings work in vision transformer from paper?

In Machine Learning "embedding" means taking some set of raw inputs (like natural language tokens in NLP or image patches in your example) and converting them to vectors somehow. The ...
Kostya's user avatar
  • 2,534
2 votes

What do we mean by the notation $\mathbf{x}_{p} \in \mathbb{R}^{N \times\left(P^{2} \cdot C\right)}$?

This itself isn't really an expression but a description of what $x_p$ looks like. Specifically, $x_p$ is a real-valued vector with the shape [N, P^2 * C]. Of ...
Chillston's user avatar
  • 1,748
2 votes

Why does CLIP use a decoder-only transformer for encoding text?

I believe because Decoder-only basically cuts down the model size in half, and has also shown empirically to be better. In the original Transformer paper, the evaluation task was about Machine ...
Minh-Long Luu's user avatar
2 votes

Does the position of the tokens in Vision Transformer matter?

It does not matter. Although, I can imagine a situation where it could matter a bit - when position embeddings are not learnt but calculated and fixed like in the original transformer (Attention is ...
hans's user avatar
  • 71
1 vote
Accepted

Vision transformer for image segmentation

Understanding segmentation heads: They are essentially the final layers of your model responsible for translating the rich feature representations from the backbone(ViT in this case) into pixel-level ...
Kulin Patel's user avatar
1 vote

Spatial vs spatiotemporal methods for object counting in low frame-rate videos

It depends. The more information you input for a sample allows the model to better understand it. However, due to differences on data quality, dataset size, model size, etc., no one can guarantee your ...
t4rf9's user avatar
  • 11
1 vote
Accepted

Can Vision Transformers be used to extract features?

Yes, of course they can be used to extract features, just like convolutional networks, even in supervised settings. ViTs are not exclusive to classification, their intermediate layers can also be used ...
Dr. Snoopy's user avatar
  • 1,345
1 vote

How "Patch Merging" works in SWIN-Transformers?

What happens during the patch merging? Concatenationating is only one part of the whole operations. Below, I quote the code from the original implementation and explain step by step what happens. <...
emely_pi's user avatar
  • 277
1 vote

What is the difference between Mean Teacher and Knowledge Distillation?

Knowledge Distillation refers to using a teacher model and distilling its knowlege to a student model, mostly done by the teacher providing soft labels for the student model to create loss. So ...
DKDK's user avatar
  • 329

Only top scored, non community-wiki answers of a minimum length are eligible