Questions tagged [vision-transformer]
The vision-transformer tag has no usage guidance.
33 questions
-1
votes
0
answers
45
views
"meta-llama/Llama-3.2-90B-Vision-Instruct" continually crashing with "torch.OutOfMemoryError: CUDA out of memory. Tried to allocate"
I'm trying to run "meta-llama/Llama-3.2-90B-Vision-Instruct" as a containerized vllm on openshift with a single NVIDIA v100 GPU (with 8 GPUs available).
The pod runs, however after about 2 ...
1
vote
0
answers
12
views
How to adopt Transformers for classifying time series of 2D images irregularly sampled across multiple filters?
I’m working on classifying time series of 2D images observed in multiple filters (or channels), but not at the same time. For example, I have observations in two filters—g-band and r-band—taken at ...
- 111
2
votes
1
answer
250
views
Insight into architecture details of Vision Llama 3.2
I would like to understand the internals of Meta's new multimodal Vision-Text 3.2 models. Not much I could find in online sources or blogs. The code is available and I'm trying to read it, but my ...
- 384
0
votes
0
answers
19
views
Should I interleave sin and cosine in sinusoidal positional encoding?
I'm trying to implement a sinusoidal positional encoding. I found two solutions that give different encodings. I am wondering if one of them is wrong or both are correct. I showcase visual figures of ...
- 101
0
votes
0
answers
105
views
Llama 3.2 Vision-Instruct Inference Speed on A100 or H100 GPU
Can anyone provide an estimated time of how long does it take for Llama-3.2 Vision-Instruct 11-B model to:
process an image size of 1-MB and prompt size of 1000 words and
generate a response of 500 ...
- 1
0
votes
0
answers
24
views
Llama-3.2-3b model finetuning for vqa loss not converging
I am trying to make a video question answering model trained on tgif qa dataset. I am using XCLIP as my video Embedding generator [ sampling 16 frames] and then my decoder is llama 3.2 3b . I am using ...
0
votes
0
answers
20
views
Getting an error importing SWin transformer using tensorflow in colab
I am trying to load swin transformer from tfhub as follows but on loading the model I get an error.
def load_model(): model_url = "https://tfhub.dev/google/swin_transformer/...
- 1
0
votes
0
answers
29
views
In a vision transformer, are the patch outputs for the last layer unused?
In a vision transformer (https://arxiv.org/pdf/2010.11929 ), it seems like the final MLP head for prediction is attached only to the last layer's [cls] token embedding (Figure 1 and Eqn 4).
Does this ...
- 136
0
votes
0
answers
30
views
Why is the CLIP model sometimes referred to by the name of its image encoder only (e.g., ViT-B/32)?
I've noticed that in some discussions, the CLIP model is referred to by the name of its image encoder only, such as "ViT-B/32." However, CLIP consists of both an image encoder and a text ...
- 3,990
1
vote
3
answers
415
views
Vision transformer for image segmentation
I am working with vision transformers (ViT) for the task of image segmentation, but I am unsure of which segmentation head to use.
I know I need a vision transformer as my backbone, and a segmentation ...
- 13
0
votes
1
answer
65
views
Fintuning Time of LoRA
I tried lora (on ViT) and i thought it would reduce the finetuning time, but it is same. is it right??
I checked that the original network's weights are forzen correctly.
Then LoRA's advantage is ...
- 13
0
votes
1
answer
23
views
Spatial vs spatiotemporal methods for object counting in low frame-rate videos
I'm currently working on an object counting/density estimation task using low frame rate video (~2 fps) in a traffic setting. I've explored a lot of literature on both spatial methods (i.e. using only ...
- 11
0
votes
0
answers
39
views
can Vision transformers be used to retain the relevant features (drop unrelated features from the clutter in image) and map to the specific query
Background, I have good understanding of ML 101 (supervised, unsupervised, tensorflow etc), however just getting into transformers & gen-AI.
I have recently started looking into Transformers/ViT ...
- 1
1
vote
0
answers
30
views
Can a transformer detect same object with different sizes?
Suppose a vision transformer has trained to detect this cat picture
Next we show it another picture of a zoomed-in cat (taken from the same image) and asked it to identify the picture
The linearized ...
- 157
0
votes
0
answers
39
views
Can the vision transformer solve object permanence?
Suppose a ball is rolling down an incline as follows:
In the middle of the path, a curtain blocks the camera view of the ball momentarily. Soon the ball reappears after passing behind the curtain.
...
- 157
1
vote
0
answers
287
views
Glass Degradation Video Prediction
We are working on the subject above, where a sequence of $n$ glass frames forms an example with an associate target that is a video of the glass future state.
We would like to know if there exists an ...
0
votes
0
answers
98
views
Where U-Net and Convolutional layers are settled in Stable Diffusion model?
When I read about Stable Diffusion model, they usually talk about adjusting convolution layers or U-Net weights. I believe they both should be related together and the U-Net is the part that accepts ...
- 101
0
votes
0
answers
30
views
How to train ViT on smaller datasets?
I know ViTs aren't made for small datasets and low resolution. But have you ever reached traditional CNN accuracy using ViT on CIFAR10/100.
I have been playing around with ViT on CIFAR10 and 100. But ...
- 21
0
votes
1
answer
615
views
Can Vision Transformers be used to extract features?
Can Vision Transformers be used to extract features, just like with VGG ?
I am interested in using this vision transformer in extracting features (https://huggingface.co/google/vit-base-patch16-224)
...
6
votes
1
answer
4k
views
Which situation will helpful using encoder or decoder or both in transformer model?
I have some questions about using (encoder / decoder / encoder-decoder) transformer models, included (language) transformer or Vision transformer.
The overall form of a transformer consists of an ...
- 69
2
votes
0
answers
65
views
Is it possible for original Vision Transformer (ViT) to do fine-grained semanantic segmentation? if so, how?
As far as I know, in the original ViT, the image is first divided to a fixed size of patch (16x16, for example) then they are flattened and treated as tokens and fed into Transformer.
Without using ...
- 21
0
votes
0
answers
211
views
How does Position embeddings work in Vision Transformer
I'm a bit confused how the position embedding in happened to each patch in the transformer. I thought Ideally we'd want each patch to have a value of (1, 2, 3, 4....) to describe the position of the ...
- 3
4
votes
2
answers
3k
views
Why does CLIP use a decoder-only transformer for encoding text?
In CLIP [1], the authors train a model to learn multi-modal (text, vision) embeddings by maximizing the cosine similarity between text and image embeddings produced by text and image encoders.
For the ...
- 329
3
votes
2
answers
825
views
Does the position of the tokens in Vision Transformer matter?
I am reading through the Vision Transformer paper and other related papers, such as DeiT and Visual Prompt Tuning (VPT). I wonder if the position of the tokens that flow through the Transformer encode ...
- 1,160
0
votes
1
answer
60
views
What do we mean by the notation $\mathbf{x}_{p} \in \mathbb{R}^{N \times\left(P^{2} \cdot C\right)}$?
I was going through this VIT paper, what will it look like in torch , if we are trying to write this expression.
- 101
2
votes
0
answers
60
views
What are the specific differences between vision transformers variants?
I have tried 4 different types of attacks on vision transformers (ViT small and tiny, DeiT small and tiny) but the attack successes on smaller versions are higher than the tiny versions. My ...
- 31
0
votes
1
answer
3k
views
How "Patch Merging" works in SWIN-Transformers?
In the SOTA paper: SWIN-Transformers, the authors have tried their best to explain everything clearly. I have got an idea of how it works except the Patch Merging ...
- 263
1
vote
1
answer
1k
views
What is the difference between Mean Teacher and Knowledge Distillation?
I recently read two papers:
BYOL Bootstrap your own latent: A new approach to self-supervised Learning
DINO Emerging Properties in Self-Supervised Vision Transformers.
I am confused about the terms ...
5
votes
1
answer
2k
views
Do Vision Transformers handle arbitrary sequence lengths the same way as normal Transformers?
Does ViT do handle arbitrary sequence lengths using masking the same way the normal Transformer does?
The ViT paper doesn't mention anything about it, so I assume it uses masking like the normal ...
- 53
2
votes
1
answer
724
views
What are the major layers in a Vision Transformer?
Currently, I am studying deepfake detection using deep learning methods. Convolution neural networks, recurrent neural networks, long-short term memory networks, and vision transformers are famous ...
0
votes
1
answer
230
views
What is the difference between a vision transformer and image-based relational learning?
I am trying to figure out the difference between the architecture used in this and this paper. It looks like both used multi-headed self-attention and therefore should be the same in principle.
- 670
8
votes
2
answers
6k
views
Why class embedding token is added to the Visual Transformer?
In the famous work on the Visual Transformers, the image is split into patches of a certain size (say 16x16), and these patches are treated as tokens in the NLP tasks. In order to perform ...
- 2,754
4
votes
1
answer
2k
views
How does the embeddings work in vision transformer from paper?
I get the part from the paper where the image is split into P say 16x16 (smaller images) patches and then you have to ...
- 263