1
$\begingroup$

I have an issue where I'm trying to use the openAI API to input a very large custom knowledge base (exceeding 1GB) that allows the user to ask questions based on that base to receive intelligent answers. However, the openAI API is very restraining in the tokens and requests that can be inputted, and I was wondering if there is another tool that is more suited for my problem. I figured it would have to utilize an LLM to understand and parse the info.

I tried different measures such as smartly looking up specific sections of info then inputting a smaller chunk it into GPT, but I'm still largely restricted by the API's restraints.

I'm quite new to this, so any ideas and suggestions would be appreciated

$\endgroup$
2
  • $\begingroup$ Perhaps this is a good case for training your own model on your knowledge base? $\endgroup$ Jul 4, 2023 at 12:12
  • $\begingroup$ The model needs to be able to make inferences between different documents in the knowledge base. My goal is to have something with the intelligence of an LLM like ChatGPT, and I don't think training my own model to do that is feasible. I'd like to know if something like that could be done though. $\endgroup$
    – Hou Wan
    Jul 5, 2023 at 3:38

1 Answer 1

1
$\begingroup$

It sounds like you need a method to determine what subset of your data is necessary for a given query. There's a lot of different methods for document retrieval. I'd start with something simple like BM25 or Tf-IDF, the sentence-transformers library also has some models you can try for semantic search.

Depending on how complex your question is, you can also have the LLM generate sub-queries and retrieve documents based on those as well.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .