GPT3 and 4 are both examples of decoder-only models. However OpenAI offers an text embedding API endpoint based on these models. This begs the general question how can one obtain text embeddings from a decoder-only transformer model?


1 Answer 1


The GPT models (as manifested using the decoder block in the original Transformer architecture) are not generating the embedding. However, the weights from the GPT are being used as the initial weights in a new model, dubbed CPT [1], that can create embeddings.

The key lies in Contrastive Modeling techniques - acquire pair data and train a new model to minimize loss on the known pairs. Look at the CLIP model for a good review of contrastive loss. The CLIP model embeds the text and image into the same vector space. Thus, CLIP and contrastive models know how to create embeddings.

Pair data, luckily, is prevalent on the web and easy to acquire: [image, caption], [docstring, code], [podcast titles, podcast descriptions], etc.

One key ingredient: you get way better results when you start with an initialized pre-trained model. That's where the GPT leverage comes in. They are using the GPT models (at various sizes like Ada, Babbage, Curie, DaVinci...) as the initial weights on a new constrastive model.

Thus, the formula is:

  1. A pre-trained GPT for initial weights
  2. Pair data for training
  3. Contrastive loss training

The resulting model is what OpenAI calls a "cpt" model.

It's unclear how well these contrastive loss embedding models work compared to a more traditional encoder-based LLM block (like universal-sentence-encode). See this blog post for more.

  • $\begingroup$ That paper uses encoder only models. How does it make sense to initialize encoders with decoder weights? $\endgroup$ Feb 18 at 23:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .