8
$\begingroup$

Are there any algorithms (or software libraries) that can be used to detect the similarity of concepts in text, regardless of articulation, grammar, synonyms, etc.?

For example, these phrases:

Outside, it is warm.

Outside, it is hot.

Outside, it is not cold.

It is not cold outside.

Should be similar to this phrase:

It is warm outside.

Ideally, the algorithm (or software) would be capable of generating a score from 0 to 1, based on the concept similarity. The goal is to use this algorithm or software to map a large number of statements to a single, similar original statement. It is for this mapping of a given statement to the original statement that the aforementioned similarity score would be generated.

Does such an algorithm (or software) already exist?

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
5
$\begingroup$

Doc2Vec

Doc2Vec comes to mind, here's the original publication. The approach has been shown to be very successful for certain NLP-based problems, though I haven't personally used it for a project yet.

There are a number of implementations of Doc2Vec. If you're using Python, one to look at is gensim.

Word2Vec

Word2Vec is similar to Doc2Vec and perhaps more in line with what you're looking for. Here's the original publication, and another publication that does a nice job explaining it further.

Tensorflow has a tutorial for setting up a Word2Vec model. Gensim also has a Word2Vec implementation.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Word2Vec (and probably Doc2Vec) can be used to learn embeddings, but are they used directly to compare "concept similarity"? Or do you need to learn the embeddings first and then use some kind of "similarity metric" between embeddings? Also, why would you learn embeddings, and what do they represent? This is not clear from this answer. So, I think you should edit this answer to clarify that. I am familiar with these concepts, so I have an idea of what the answers to my questions here are, but I suspect that, in this form, this answer doesn't directly answer the original question. $\endgroup$
    – nbro
    Jan 23, 2022 at 10:25
  • $\begingroup$ @nbro I wrote this answer five years ago. While I stand by it for the the reasons that 1. Word2Vec embeddings enable one to do concept arithmetic over learned embeddings, and 2. all the examples described in the OPs question would be embedded closely to one another in a Word2Vec embedding (and this distance could be measured and normalized to the range [0,1] as needed), I suspect that there are better solutions now. If you feel these two points aren't sufficient explanation for the community, feel free to add your own answer with an update. $\endgroup$
    – Greenstick
    Jan 24, 2022 at 17:28
0
$\begingroup$

In machine translation, there is a widely used BLEU score ( https://en.wikipedia.org/wiki/BLEU ). It simply counts the matching n-grams between two segments of text and returns a 0-1 score based on that.

The problem with this method is that it would give the same score to pairs "It is hot"/"It is cold" and "It is hot" / "It is warm". There is no nuance for spelling or synonyms: for each word, you either have a literal match or you don't.

A recent refinement to BLEU is BLEURT (https://arxiv.org/abs/2004.04696). The key idea is to also look at BERT embedding for both segments of text, which now allows you to notice that "hot/warm" are more similar than "hot/cold", and produce a more nuanced score.

This is all specific to machine translation evaluation , and may or may not work for your specific application.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .