Why can't Lucene search be used to power LLM applications?

Question

w.r.t. LLM applications using the RAG (retriever-augmented-generation) architecture, people have started taken it for granted that it will be powered by a vector database. e.g., see this:

The most important piece of the preprocessing pipeline, from a systems standpoint, is the vector database.

Why can't lucene index (full-text search) be used for the retriever? Is there any objective study that has been done comparing quality of results using full-text search vs. using a vector database?

As I was writing this, even lucene seems to have jumped on the vector bandwagon. see this

Answer 1

At least the traditional Lucene full-text search is not vector based but an inverted index structure which can sort document relevance ranking via scoring functions such as the term frequency-inverse document frequency (TF-IDF) or the Okapi BM25 algorithm. On the other hand vector-based LLMs often use techniques such as cosine similarity instead for the new semantic relevance ranking which the traditional full-text search cannot satisfactorily achieve.

Answer 2

They are used. Tf-IDF and BM25 are pretty strong baselines for document retrieval, and papers on semantic search often include these methods as a comparison point.

e.g., papers like this one on open domain QA retrieve documents based on the question being asked. Vector representations perform better, but BM25 is still pretty solid, especially when performance is a bottleneck.

Answer 3

Because Lucene does not do a comparison of MEANING. It is, essentially, still a text search.

Vector databases are an attempt to find similar MEANING, even if the words are not identical. That is a totally different area of search. A standard vector (going by OpenAI here) has more than 1500 dimensions and identifies the similarity based on every dimension - Lucene would not even know the difference between "large village" and "city" while in reality they are QUITE similar. Or would not know that a "House" is close in meaning to a "Building".

Is there any objective study

No one reading the documentation would need that - they simply do too many different things. Heck, Lucene does not even HAVE semantic search built in. The difference is not in the search - it is in what is searched and how the vector for a vector database is constructed to try to capture the meaning, not the similarity in words. At the minimum you would need Lucene + a semantic search mechanism - and then you would have to calculate something like a similarity along a lot of axis. Also known as a vector.