I'm building a RAG pipeline to extract real estate phrases from excel documents. These phrases are short (2-5 words) and are often phrased differently.

I've manually added in different phrases to each prompt to suggest the LLM to look for them.

What else could I do? What more powerful options do I have?

  • Search for each term manually? -> slow due to loads of calls
  • Embed the terms in the excel doc and similarity search the embedding of 'Total Income' against those in the excel doc?
# Example prompt with 4 similar terms
income = f"""What is the Total Income? 

It may also be called: 
- Total Revenue
- Effective Gross Income
- Total Operating Receipts
- Total Operating Income

Or something similar

Return a signed number or nan. Do not write anything else."""
  • 1
    $\begingroup$ If you know what keywords you're looking for, it might be easier to build an inverted index for keyword searching $\endgroup$
    – Karl
    Oct 5, 2023 at 2:01
  • $\begingroup$ @Karl can you please expand? What does building an inverted index involve? How would I do it? Code sample? $\endgroup$
    – codeananda
    Oct 6, 2023 at 13:05
  • $\begingroup$ It's a database structure for fast text keyword searching en.wikipedia.org/wiki/Inverted_index elastic.co/elasticsearch $\endgroup$
    – Karl
    Oct 6, 2023 at 19:15


You must log in to answer this question.