0
$\begingroup$

I'm a bit confused how the position embedding in happened to each patch in the transformer. I thought Ideally we'd want each patch to have a value of (1, 2, 3, 4....) to describe the position of the patch in the image. but from the implementation here there do something like this:

 # positional embedding
            self.pos_embed = nn.Parameter(
                torch.zeros(1, num_patches, embedding_dim)
            )

Which is quite confusing because now we have some sort of mapping instead of just a value appended to each patch. Also, there is some sort of implicit position appended to the patch right? Assume we have a patch embedding output (1, 256, 768); corresponding to (batch, num_patches, position_embedding). since we have 256 patches, then can't our network understand that each patch is in the position of its index value? Why do we have to explicitly define a position embedding for each patch?. Also, please kindly explain the implementation above I'm not sure I understand the mapping and why its initialised to zero

Improve this question
$\endgroup$
3
  • $\begingroup$ Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. $\endgroup$
    – Community Bot
    Commented Apr 3, 2023 at 19:29
  • $\begingroup$ the initialization is likely done somewhere else, the zeros in your linked code just reserve memory for the real parameters later. $\endgroup$
    – N. Kiefer
    Commented Apr 6, 2023 at 9:58
  • $\begingroup$ Can you please put your specific question in the title? $\endgroup$
    – nbro
    Commented Apr 13, 2023 at 20:32

0

Reset to default

You must log in to answer this question.