Understanding embedding outputs in transformer models like CLIP

Question

I'm working with OpenAI's CLIP model and trying to understand the output of the text encoder. When I input a short prompt like "cat", the output is a tensor of shape [77, 1024]. My understanding is that the 1024 represents the dimensionality of the embeddings, and the 77 represents the maximum sequence length that the model can handle.

Given that "apple" would be tokenized into far fewer than 77 tokens, I'm assuming that the remaining tokens are padding tokens. However, when I inspect the tensor, I don't see any zero values. I was expecting the embeddings for the padding tokens to be zero vectors, but this doesn't seem to be the case.

My current hypothesis is that only the first few 1024-dimensional vectors in the tensor (corresponding to the tokens in my input) are significant, and the remaining vectors (corresponding to padding tokens) do not carry meaningful information about my input. Is this understanding correct?

Also, could someone explain why the embeddings for the padding tokens are not zero vectors? How does the model ensure that these padding tokens do not contribute to the output?

Answer 1

Usually the embeddings are not zero for the padding, since they are part fo the lookup table of the embedding layer, however in addition to the input, a mask is forwarded through the forward pass of the network to mask them out when it's relevant (in transformers encoder used in CLIP, in the cross-attention)

So the actual transformer attention is something like: $$ A = softmax(\frac{QK^T}{\sqrt{d_k}} \cdot (MM^T\cdot -\infty))V $$

where $M$ is a vector $1xN$ ($N$ being the length of the sequence you have inputted, with $0$ for actual tokens, $1$ for masking)

By doing so, you are adding $-\infty$ to the padding tokens, thus the attention will be 0, which means that in the last product ($softmax \cdot V$) those embeddings that you are seeing not being zero are not considered

However, I want to highlight that if the padding tokens were zeros, this still is needed, as after normalizatio, attention or FFNN step, they might not be zero anymore, thus contributing to the whole next attention step, which would make the network no more length invariant