I've been studying retrieval augmented generation and vector databases recently. In a nutshell, vector based retrieval works by first dividing up input data(whether it's a string, image, etc) into chunks and then generating a vector embedding for each of those chunks. The key is to choose a chunk size that is not too large nor not to small so that the LLM can query the accurate data to augment its response.

This is relatively straightforward. I wonder if entire LLMs can share each other's knowledge as of current technology.

For instance, say I have an LLM that is really effective at processing text prompts and returning text answers, another LLM that is really effective at generating images given a text, and another LLM that is really effective in analyzing images or visual input. Instead of merely vector embedding the chunked outputs of each LLM such that the text based LLM will use the embedded user response to query each of the vector embeddings to augment its final response, I wonder if there is a way to "merge" the 3 LLMs in a way such that they are separate, but one. So whatever one LLM can do must be able to be done by other LLMs. This method of information retrieval between LLMs will not only allow them augment their responses with static - vector embedded - data, but also allow them to augment their responses with dynamic data that another LLM has generated or is in the process of generating.

Is there any literature, latest technology, frameworks, discussions, etc on this topic?

// When I say effective, I mean these things: // 1. How specific the LLM is when understanding the user prompt (e.g. if the user asks the LLM to generate a poem about recursion by making the poem recursive itself, the LLM should attempt to incorporate the latter part: "making the poem recursive itself" instead of ignoring parts of the prompt it is not trained on or less capable of processing. // 2. The quality of the generated output after processing the user prompt. This is straightforward. Is the image garbage? Is a textual response incorrect or deviating from the user's instructions, despite 1. being fulfilled?



You must log in to answer this question.