Named entity recognition (NER), also known as entity chunking/extraction, is a popular technique used in information extraction to identify and segment the named entities and classify or categorize them under various predefined classes.
Briefly, how does NER work? What are the main ideas behind it? And which algorithms are used to perform NER?
There are different algorithms, each with their advantages and disadvantages.
Gazetteers: these have lists of the entities to be recognised, eg list of countries, cities, people, companies, whatever is required. They typically use a fuzzy matching algorithm to capture cases where the entity is not written in exactly the same way as in the list. For example, USA or U.S.A., United States, United States of America, US of A, etc. Advantage: generally good precision, ie can identify known entities. Disadvantage: can only find known entities
Contextual Clues: here you have patterns that you find in the text, eg [PERSON], chairman of [COMPANY]. In this case, sentences like Jeff Bezos, chairman of Amazon, will match, even if you have never come across either Bezos or Amazon. Advantage: can find entities you didn't know about. Disadvantages: could end up with false positives, might be quite labour-intensive to come up with patterns; patterns depend on the domain (newspapers vs textbooks vs novels etc_
Structural description: this is basically a 'grammar' describing what your entities look like, eg (in some kind of pseudo-regex): title? [first_name|initial] [middle_name|initial]? surname would match "Mr J. R. Ewing" or "Bob Smith". Similar descriptions could match typical company names; you'd probably still need lists of possible surnames or firstnames. Advantages: some flexibility, and potentially good precision. Disadvantages: patterns need to be developed and maintained.
Ideally you would want to combine all three approaches for a hybrid one to get the advantages of recognising unknown entities while keeping excess false positives in check.
There might also be other machine-learning approaches, but I'm not too familiar with those. The main problem is that they are hard to fine-tune or work out why they do what they do.
UPDATE: A good starting point would be to use a gazetteer-based approach to annotate some training data, and use that to identify contextual patterns. You can then use that data to train a machine learning approach (see OmG's answer on CRF) to broaden the approach; and then you add newly recognised entities to your list.
Ideally you would want to have a gazette as your main database to avoid false positives, and use machine-learning or contextual patterns to just capture previously unseen entities.
One of the renowned learning algorithms for NER tagging is the conditional random field (CRF). As you can see in the provided link, sequence labeling algorithms such as RNN with LSTM can be used to named entity recognition as well. By the way, you can find an implementation of the CRF for NER tagging in this source.
Notice that, the method of providing training data can be helpful to pass the data into the standard libraries of CRF without any extra preprocessing. One of the standard methods is BIO method(B (Begin), I (Interior), and E (End)). You can find more about it in this post.
