I'm doing a paper for a class on the topic of big problems that are still prevalent in AI, specifically in the area of Natural Language Processing and Understanding. From what I understand, the areas:

  • Text classification
  • Entity recognition
  • Translation
  • POS tagging

are for the most part solved or perform at a high level currently, but areas such as:

  • Text summarization
  • Conversational systems
  • Contextual systems (relying on the previous context that will impact current prediction)

are still relatively unsolved or are a big area of research (although this could very well change soon with the releases of big transformer models from what I've read).

For people who have experience in the field, what are areas that are still big challenges in NLP/ NLU? Why are these areas (doesn't have to be ones I've listed) so tough to figure out?


Edit 12/9/2019: For anyone who wants to put an answer / opinion, please do. I feel this question benefits from a lot of different perspectives and as new technologies arise answers will be different.

  • $\begingroup$ The first step in answering the question would be to collect and summarize reports about failed projects from within Natural language processing. What was mentioned often in the press is the IBM Watson flagship project, which was announced as a universal language parser and generator but wasn't successful in convincing the end user. It seems, that NLP works fine in the clean lab but in the normal world it struggles in simple tasks. $\endgroup$ – Manuel Rodriguez Dec 3 '19 at 21:01
  • $\begingroup$ Thanks for the reply @ManuelRodriguez, I'm more curious though on what are current challenges/ research areas rather than how to do my research, although still appreciated. $\endgroup$ – Landon G Dec 3 '19 at 21:18
  • $\begingroup$ @LandonG Do not ask for opinions, but for facts, if possible. You should ask "What are the current challenges in NLP and NLU and why?" and not "In your opinion, what are the current challenges in NLP and NLU?". $\endgroup$ – nbro Dec 9 '19 at 20:51
  • $\begingroup$ @nbro I agree with you but to be fair, the NLP field is pretty broad and I don't think asking for an educated opinion is necessarily a bad thing in this case. It's hard to ask for concrete facts about something that is developing rapidly ( plus if someone disagrees it could spark a debate). Isn't this whole site essentially backed on opinions and advice besides the technical questions anyways? Just an opinion ;) $\endgroup$ – Landon G Dec 10 '19 at 2:57

According to a nice article by Sebastian Ruder https://ruder.io/4-biggest-open-problems-in-nlp/ based on answers from top NLP researchers https://docs.google.com/document/d/18NoNdArdzDLJFQGBMVMsQ-iLOowP1XXDaSVRmYN0IyM/edit

  1. Natural language understanding
  2. NLP for low-resource scenarios
  3. Reasoning about large or multiple documents
  4. Datasets, problems, and evaluation

I recommend having a look at the article. More details in the slides https://drive.google.com/file/d/15ehMIJ7wY9A7RSmyJPNmrBMuC7se0PMP/view


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.