Why does CLIP use a decoder-only transformer for encoding text?

Question

In CLIP [1], the authors train a model to learn multi-modal (text, audio) embeddings by maximizing the cosine similarity between text and image embeddings produced by text and image encoders.

For the text encoder, the authors choose to use a variant of GPT2 which is a decoder-only transformer, taking the activations of the highest layer of the transformer at the [EOS] token the feature representation of the text (emphasis mine):

The text encoder is a Transformer (Vaswani et al., 2017) with the architecture modifications described in Radford et al. (2019). As a base size we use a 63M-parameter 12- layer 512-wide model with 8 attention heads. The trans- former operates on a lower-cased byte pair encoding (BPE) representation of the text with a 49,152 vocab size (Sennrich et al., 2015). For computational efficiency, the max sequence length was capped at 76. The text sequence is bracketed with [SOS] and [EOS] tokens and the activations of the highest layer of the transformer at the [EOS] token are treated as the feature representation of the text which is layer normalized and then linearly projected into the multi-modal embedding space.

I found this pretty weird considering that they could have used an encoder (a-la BERT) which to me seem more fitted to act as encoders than decoders. Perhaps they wanted to enable generative text capabilities, but they could've achieved that with an encoder-decoder architecture (a-la T5) too.

I was expecting ablations on the text-encoder architecture, motivating their choices, but found none. Any clue why they made these choices?

References:

[1] A. Radford et al., ‘Learning Transferable Visual Models From Natural Language Supervision’, in Proceedings of the 38th International Conference on Machine Learning, Jul. 2021, pp. 8748–8763. Accessed: Feb. 07, 2023. [Online]. Available: https://proceedings.mlr.press/v139/radford21a.html

Answer 1

I believe because Decoder-only basically cuts down the model size in half, and has also shown empirically to be better.

In the original Transformer paper, the evaluation task was about Machine Translation, which at that time, encoder-decoder architecture was very successful.

This paper is probably the first to propose the decoder-only Transformer, in which they observe the following improvements:

• It removes the encoder, which means half the parameters and hyperparameters.
• It helps them with long input sentences
• The inputs of the Encoder and Decoder are the same, so basically it is probably redundant

Later paper also finds out that decoder-only works better than encoder-decoder part. One thing to note about this is Encoder is Bi-directional while Decoder is Uni-directional. This nature fits with GPT-2, which is an autoregressive language model.

Related:

• This paper may be related. It studies different decoder-encoder variants and finds that decoder-only performs better.
• A StackExchange question regarding the decoder in BERT.

Answer 2

I think one potential explanation for their text-encoding method choice is that they probably realised that an image will be typically captioned with more than one word (which on its own can be multiple tokens), so they need sentence-embeddings of the text, not token embeddings which is what you get with BERT. Basically, image::sentence.

You notice this when using the CLIP encoder in code - it gives you sentence embeddings, not token embeddings.

Because they use a decoder-only transformer, which is autoregressive, they can take the feature representations at the EOS token as sentence representations of the sentence that preceded the EOS token.

Had they used an encoder, they would've had think about some aggregation method to go from token embeddings to sentence embeddings (Although, to be fair, this may have been as simple as using the CLS token representation).