Are there any algorithms or software libraries that can be used to detect the similarity of concepts in text, regardless of articulation, grammar, synonyms, etc.?

For example, these phrases:

Outside, it is warm.

Outside, it is hot.

Outside, it is not cold.

It is not cold outside.

Should be similar to this phrase:

It is warm outside.

Ideally, the algorithm or software would be capable of generating a score from 0 to 1, based on the concept similarity. The goal is to use this algorithm or software to map a large number of statements to a single, similar original statement. It is for this mapping of a given statement to the original statement that the aforementioned similarity score would be generated.

Does such an algorithm of software already exist?

  • $\begingroup$ I kindly request you to post this question in cross validated community $\endgroup$
    – quintumnia
    Sep 26 '17 at 10:05
  • $\begingroup$ Cross Validated would be the place for more complex followup questions. However this question is quite basic, and therefore useful in terms of fundamental concepts and techniques related to this type of problem for the wider AI community. Welcome to AI! $\endgroup$
    – DukeZhou
    Sep 26 '17 at 21:30
  • $\begingroup$ "what the sentence owner mean?", isn't it NLP's main focus indirectly? :) $\endgroup$
    – ibubi
    Sep 27 '17 at 5:47
  • $\begingroup$ Unfortunately, I think this may be a bit too broad. Asking for a list of resources seems like it will have too many possible answers, and so not a good fit for Stack Exchange's Q&A model. (cc @DukeZhou) $\endgroup$
    – Mithical
    Sep 27 '17 at 14:24


Doc2Vec comes to mind, here's the original publication. The approach has been shown to be very successful for certain NLP-based problems, though I haven't personally used it for a project yet.

There are a number of implementations of Doc2Vec. If you're using Python, one to look at is gensim.


Word2Vec is similar to Doc2Vec and perhaps more in line with what you're looking for. Here's the original publication, and another publication that does a nice job explaining it further.

Tensorflow has a tutorial for setting up a Word2Vec model. Gensim also has a Word2Vec implementation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.