1
$\begingroup$

Since the context/memory of a chat or question for LLMs more precisely GPT is limited to a token length I struggle about how to provide own data that the model got not trained on. A very common approach looks like embeddings are the way to.

OpenAI provided an article https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb how to create an embedding of a user query, match it against a local vector database ans provide the closest results as text to the context/memory.

Here I do struggle, since it might be very well possible that even we find the most matching documents locally in a vector database, context might still be too small if we would like to provide multiple matches.

The question to me is, how could I send all the relevant embedding vectors rather than the relevant texts which got matched to the vectors? These vectors are highly condensed and would save a lot of tokens. GPT would anyhow be able to understand the vector since they created it from their embeddings API, right?

Or is it just not possible to convert the vector back to text at their end?

$\endgroup$

2 Answers 2

2
$\begingroup$

Do you mean sending the vector of embedded text as the context, instead of the text itself?

If you think it through that might mean sending in [2.12234, 12.134123, 3.132412, ....., 1.123124] to chatGPT instead of Hi, can you tell me what is the grandfather paradox?. Do you spot the problem here?

GPT models are designed in a way to only take textual inputs and spit out textual inputs. So, in short, what you are proposing is not possible with the API endpoints currently provided by OpenAI.

Theoretically, taking in an embedding vector and producing the relevant text is a possibility but considering the transformer architecture, there is a logical fallacy in your proposed argument.

  1. Embeddings are produced from the text itself, so to infer a sequence from a generated embedding would mean that you have already used up a certain toke length, i.e. at this point it really doesn't matter what is the max allowed token length.

Not just that, but the ADA embedding model is different and is fine-tuned to generate better similarity rather than aid in generating CHATesque text. So, even if OpenAI offered that possibility (which it doesn't) the generated text wouldn't be as good as sending in text as context to GPT4 api endpoint.

$\endgroup$
0
$\begingroup$

Vectors are not provided as context - you need to provide the original text. Embeddings are used with e.g. vector databased to provide similarity queries so you can reduce the context you provide to GPT by only selecting relevant parts (e.g. 1-2 documents from a library of 1000s of documents).

$\endgroup$
1
  • $\begingroup$ Pleast break the answer into smaller bits for unexperienced questioners. It will increase your answer's effectiveness. $\endgroup$
    – Chinmay
    Jul 22, 2023 at 18:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .