Suppose a vision transformer has trained to detect this cat picture

enter image description here

Next we show it another picture of a zoomed-in cat (taken from the same image) and asked it to identify the picture

enter image description here

The linearized patches will look very different for the first picture and the second picture.

In fact, any sub-sequence from the first picture will look very different from the second picture. For example, there will be many more black patches in the second picture because of the sheer size of the cat compared to the first picture with the tiny cat.

Can the transformer then recognize the second picture as "cat" entirely based on the smaller/scaled version of the cat it has trained on?

How would attention find the scaling similarity between two pictures/prompts?



You must log in to answer this question.

Browse other questions tagged .