I am using openai's
text-embedding-ada-002 embeddings model to do a semantic search on a database of articles to find articles that are most related to a given input text. I am looking for a way to define a minimum similarity score to prevent returning articles that aren't actually related enough.
There is two difficulties that I have:
- For some search queries a certain similarity score seems appropriate as a minimum treshold value, but then for others that minimum value seems to be too strict. For instance I find that for very well defined specific topics you generally want a higher treshold similarity score than for more broad or generic texts. That's my intuition so far at least.
- The scores of the openai embedding model almost always fall between 0.77 and 1 instead of using the entire range of -1 to 1 and in reality the scores in normal cases all fall around 0.88. Having all scores so close to eachother makes it harder to pinpoint a good treshold value.
Are any known methods for determining a good treshold value for cosine similarity scores?