0
$\begingroup$

I'm using Ollama to run llm's. I can create embeddings for text and images, which I store in ChromaDB. The goal of all this, is to find content which best fits a question, so I can create a good answer.

Here is how I create embeddings for text

curl http://localhost:11434/api/embeddings -d '{
    "model": "llama2",
    "prompt": "Here is an article about llamas..."
}'

and for images (base64)

curl http://localhost:11434/api/generate -d '{
  "model": "llava",
  "prompt":"What is in this picture?",
  "stream": false,
  "images": ["iVBORw0KGgoAAAANS.....]
}

However, now I have some content which is text and image together. The text (or the image) alone is meaningless, but together they tell a complete story. So my question is how should I handle this? Is it possible to create a single embedding for both?

$\endgroup$

1 Answer 1

0
$\begingroup$

Translate the image into a description. Add the description of the image with the text.

Embed both of those so you have the information from the image and the information from the text.

How does this sound?

$\endgroup$
2
  • $\begingroup$ So with into a description you mean the process of my second code example? $\endgroup$ Commented Apr 23 at 11:12
  • $\begingroup$ Of course I could add the text in the process of creating the image embedding $\endgroup$ Commented Apr 25 at 13:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .