I've just started learning natural language processing from Dan Jurafsky's videos lectures. In that video, minute 4:56, he is stating that dialogue is a hard problem in natural language processing (NLP). Why?
$\begingroup$
$\endgroup$
0
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Dialogue is a hard problem because it requires pretty advanced cognitive functions. Leaving aside all the lower levels of language analysis (phonology if dealing with speech, morphology and syntax), you quickly run into interpretation problems that require a lot of world knowledge.
Simple question and answer is fine, and restricted domains are somewhat easier as well. As soon as you get into a normal conversation, you will refer back to things you said before, so an NLP system would have to recognise that and resolve the reference accordingly. Typically in a conversation you would use variations of reference terms: the first time you mention an object you might describe it fully, but subsequently you will use shorter terms to refer to it.
There is also a structure to conversation. This is typically modelled as conversational moves, and usually moves will have corresponding response-moves. For example, a common sequence would be greeting - greeting. Then you might have question - response - feedback. This sounds fairly easy, but once you try to annotate a dialogue with such moves you will find that it is pretty hard. As far as I am aware, there is no 'grammar' equivalent of describing the structure of conversations.
Often, pragmatic meaning is interfering with the 'surface' meaning of utterances. A statement can actually function as a question, or a question can be a command (or a statement). The pragmatics of utterances depend on the context and also the relationship between the interlocutors. If I talk to my manager, I will use language differently than when I talk to my children.
Dialogue/conversations are hard to analyse. Because of that, descriptive frameworks are still fairly limited. You need to keep track of what has said before, as that can change the way an utterance has to be interpreted. Grammatical analysis is a fairly solved problem, and word sense disambiguation as well. But pragmatics and conversational structure are still on the bleeding edge of linguistic research; at least they were when I was still teaching Discourse Analysis at university a few years ago.
For that reason, chatbots are generally not very good. Sometimes they can fool people into believing they are human speakers, but this is usually done through trickery ("smoke and mirrors") rather than competent handling of conversational structures. It's all in the little box in nbro's answer labelled "DM"...
$\begingroup$
$\endgroup$
First of all, I am not very familiar with details of NLP and NLU systems and concepts, so I will provide an answer based on the slides entitled Natural language understanding in dialogue systems (2013) by David DeVaul, a researcher on the topic.
A dialogue system is composed of different parts or modules. Here's a diagram of an example of a dialogue system.
Each of these modules can introduce errors, which are then propagated to the rest of the pipeline. Of course, this is the first clear issue of such a dialogue system. Other issues or challenges include
ambiguity of natural language (and there are different types of ambiguity, i.e. see slide number 5),
synonyms (i.e. the dialogue system needs to handle different words or expressions that mean the same thing),
context-sensitivity (i.e. the same words or expressions can mean different things in different contexts)
semantic representation (i.e. how to represent semantics)
spontaneous speech (i.e. how to handle stuff like "hm", pauses, etc.)