Speech is a major primary mechanism of communication between humans. With respect to artificial intelligence, which signal is used to identify the sequence of words in speech?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
The signal path in speech recognition as one travels further from the basic representations of sound depends increasingly on the role the recognition plays. Consider these roles.
- Translation to another language
- Comprehension as in listening to a lecture
- Comprehension as in a bunch of people out for food, drink, and laughs
- Transcription into text
There are similar differences in other natural language facilities of reading, writing, typing, speaking, interpreting body language, and expressing with body language. With auditory language, the stages of sensory processing proceed in a particular order of signal types, each representing the dynamics of language at successively abstract levels, essentially reversing the process of speaking.
- Physical dynamics of air
- Frequency dependent vibrations of cochlear hair
- Spectra, transients
- Pitch, tone/timbre, impacts, loudness
- Phonetic elements
- Linguistic elements distinct from other noise
- Beginnings of attention based functionality
The first six elements are common between all language oriented hearing. The seventh is where the above four roles diverge. In translation and comprehension, comprehension of linguistic nuance is required. Colloquialisms, euphemisms, culturally dependent references and analogies and other higher level linguistic and social constructs is required for full proficiency. For transcription, comprehension is required only to improve transcription reliability and to distinguish between phononyms like to, too, and two.
Notice that the sixth level is not words or sentences. Words and sentences are later developments in human language and occur later in an individual's childhood. Our educational systems are largely word oriented, but people do not talk or listen in words. There is no signal for "way" in the brain when someone hears, "No way!" which, in the current semantic state of U.S. English is a single linguistic element that means, "What you just said is very surprising to me." Two written words represent this one linguistic element.
Conversely, there are two distinct signal representations for, "wanted", specifically, "want," and "-ed." The second of the two is reused for, "planted." The -ed ending is not relearned for every verb.
Consider these lines out of a dialog.
Jenna: Cats run.
Chelsea: That's ridiculous.
In transcription, the voices must be distinguished and the words must be written, including the contraction of, "That is," whereas the plural of cats could be possessive, so the computer attempting this transcription must somehow chose the plural over the possessive in this case. Out of context, the pronoun, "That," does not refer to the cat or a person named Cat but the idea of the running. Consider this conversation between people out for food, drink, and laughs.
Catherine: He's so sexy.
Jenna: Cat, you always pine over the really toxic guys.
Catherine: No I don't.
Jenna: What about that cute (and well dressed) guy that wanted to meet you for lunch last week?
Chelsea: She gave him her number.
Julia: Yea, with the last two digits transposed.
Chelsea: All the Jennas I've ever met are first to run when a nice guy comes around.
Catherine nods. (beat) Patrick comes to the table.
Patrick: Hey, you're Catherine, from Central High, right? Didn't we have study hall together in junior year?
Catherine puts some cash on the table for the drinks.
Catherine: I didn't go to Central. I went to East Barnard. (beat) Girls, I'll see you later. Gotta go so I can get up for work tomorrow.
Catherine exits. Patrick wanders off.
Jenna: Cats run.
Chelsea: That's ridiculous.
Julia: What, you think she left because of work?
Chelsea: Why not?
Jenna: She works at a restaurant closed on Mondays.
This dialog is a series of signals relying heavily on understandings that are not in the language. There is no formal grammar that could faithfully guide parsing by words. The signals after all seven levels of processing still fall far short of complete descriptions of the scenarios or the ideas of the individuals involved. Yet a series of themes and triggers to existing information shared in common between those at the table communicate volumes of information.
Level six from the layers listed above is important here because people are talking and possibly loudly at other tables. Level seven of the listening is entirely different at this girl's night out and may not share any significant functionality with the listening process when Jenna, Chelsea, Catherine, and Julia are in class taking notes.
For computers to partake in a conversation like this one and actually socialize, many more functions beyond the seven layers of abstraction above must be assembled and must learn all the nuance and information brought in as cultural reference. For example, there are several cultural ideas involved.
- Some people tend to pick those that are not particularly healthy for mates.
- Getting a phone number is a first step toward talking in private.
- Cat is Catherine.
- The conversation is occurring on Sunday, thus the next morning is Monday, the day when the restaurant at which Cat works is closed.
- A man producing some connection from the past is a segue into further conversation based on some relational interest.
answered Dec 8, 2018 at 8:35
$\begingroup$
$\endgroup$
0
Not sure I understand what you mean by signal. However, incoming audio must be cleaned and processed for a machine to understand. Some people speak faster and others slower. First thing is to sample in the incoming audio. In other words use analog electic signals and store them as binary values at a set rate of sampling frequency.
Now that it is sampled we need to clean it up and extract only the voice data. Effectively the sample is a set of values (frequencies) recorded over time. Or a signal :). We can cut up this signal into many smaller components of a few milliseconds say 10ms. Each of the millisecond samples is further processed A Fourier transform is generally used to break a signal into components, or frequencies that make up the signal. This is basic signal processing and we have not even entered machine learning yet.
Now we have a fingerprint of each 10ms sample these samples are then fed into a neural network. Samples are broken between frequency drops to create blocks of sound which may or may not be a word. These blocks are passed to the neural network.
A traditional feed forward neural network will not work as it forgets the data. So a recurrent neural network is used. To train the neural network known words are tagged and encoded as a number. The neural network will output a number and the difference between this number and the expected number is the error. The error is then used to update the weights
Once we have the words, those can be further processed by other networks that specialise in natural language understanding
