By means of parts of speech tagging, words of a given sentence can be assumed to be noun/verb etc, but if the sentence is for instance:

"My favourite book is harry potter and the prizoner of azkaban"

note that the inputs I receive would be from a chat interface so having a fixed format for the data can't be expected. Is there a way to identify "harry potter and the prizoner of azkaban" as a proper noun from such messages?

Currently this query tags as:


I would like to know if this can be handled some way, or if there is another algorithm that can handle this?


I guess your problem is a form of NER (Named-Entity Recognition) tagging. NER tags consist of PER(person), LOC(location), ORG(organization) and MISC(miscellaneous) and O(other). with the help of a NER-tagger algorithm, you probably would have:

My(O) favorite(O) book(O) is(O) harry(B-MISC) potter(I-MISC) and(I-MISC) the(I-MISC) prisoner(I-MISC) of(I-MISC) azkaban(I-MISC).

now you can tokenize your text with identifying different NER tags, and join "B-" and "I-" prefixes with the same NER tag. e.g. you have "harry potter and ..." as a single token, which is a MISC and starts from harry (since harry is B-MISC) and ends to azkaban (since azkaban has the last consecutive I-MISC tag). Now you can let your POS tagger act the "harry potter ..." as a single token, and it must tag it with "NN"

Another solution for this problem is called "chunking". It works based on a set of rules and detect Noun Phrases (NPs). Here, you define a rule and create the regexp statement for it. e.g. you define all consecutive NNs as a single NP (harry potter): <NN>+ or a DT followed by a string of NNs (the brown fox): <DT>?<NN>+. Now you tokenize your text based on a regexp matching method. but in your case, chunking such a long NP is almost impossible. because a rule that finds this NP as a NP candidate, would by mistake find a lot of other string of words as a NP candidate too, while they're not a NP at all. (see nltk: information extraction)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.