1
$\begingroup$

I have many text documents and I want to identify concepts in these documents in an unsupervised manner. One of my problems is that the concepts can be bigrams, trigrams, or even longer.

So, for example, out of all the bigrams, how can I identify the ones that are more likely to represent a concept?

A concept could be "machine learning".

Are you aware of any standard approaches to solve this problem?

Edit: The corpus I am working with consists of papers accessed from web of science. That is, they are all in some given domain niche. I want to extract words, bigrams, trigrams... that represent common concepts/buzzwords from these papers. These could be "Automated machine learning", "natural language processing" et cetera. I need to be able to distinguish these from other common n-grams such as "New York", "Barack Obama",...

I know that I could do this using a NER approach but this would require hand-labelling. Are you aware of any unsupervised ways to approach this problem? Or even a semi-superised method with little labelled data?

Improve this question
$\endgroup$
2
  • $\begingroup$ I think it's important that you define what you mean by "concept", even though you provide an example. Is it a noun or a name? Are all nouns concepts? And I understood that concepts can be composed of more than one word. I think you want to exclude adjectives, adverbs, and verbs, but you should clarify this. $\endgroup$
    – nbro
    Jul 7, 2020 at 22:35
  • $\begingroup$ @nbro, thank you for the comment. I added a clarification that hopefully makes it clearer. $\endgroup$
    – Haffi112
    Jul 8, 2020 at 11:39

1 Answer 1

Reset to default
0
$\begingroup$

One of the renowned methods is using the TF-IDF method. You can define a lower bound for the TF-IDF score of the bigram or trigram to identify it as a keyword. Here, you can find more information about it.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Isn't tf-idf only used to rank words in a document with respect to all others in that document and in that database? How can you use tf-idf to actually find "concepts"? It's true that the OP did not define "concepts" though, but only gave an example. Maybe tf-idf is sufficient, but I think it's better if we ask for clarification about what a concept is. $\endgroup$
    – nbro
    Jul 7, 2020 at 22:32
  • $\begingroup$ @nbro As I know here the concept means a meaningful bigram or trigram that can be happened in the meaningful number of times. Hence, TF-IDF can be a good score to find these meaningful concepts. I agree that TF can be meaningful measure for bigrams or trigrams as well. $\endgroup$
    – OmG
    Jul 8, 2020 at 13:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .