0
$\begingroup$

Some background: I'm an EE major and data science minor, so I have a basic understanding of machine learning - I had a one semester course on it where we covered some of the most commonly used algorithms such as Naive Bayes' and K-nearest-neighbors, as well as the basics of how artificial neural networks work. I've also had EE classes on signal processing, so I'm familiar with the Fourier transform and the like. So I have a general idea of how different algorithms can, for lack of a better term, "pick up on" patterns in the training data and make predictions on the testing data, which will hopefully be reasonably accurate. And I have a general idea of how signals can be represented as discrete binary sequences which could then be used to train an ML model.

What I want to know is, what sorts of patterns, in the mathematical sense, in audio recordings of speech specifically, do these algorithms find? e.g., What allows the Alexa app to both recognize when I'm speaking vs when it's my grandma speaking, and yet still understand the same commands from both of us? What, mathematically, makes, e.g., a recording of me saying "Alexa play music" both distinguishable from a recording of a different person saying the same command, and yet still recognizably contain the same information of the command? In other words, how do the algorithms extract and match the speaker-specific and content-specific information from the speech signals?

Improve this question
$\endgroup$
2
  • $\begingroup$ Is there a reason you find this odd? After all, you understand yourself, your grandmother, etc. $\endgroup$
    – Dave
    Commented Jul 5, 2023 at 22:18
  • 1
    $\begingroup$ @Dave, it's not that I find it odd -- it's just that I don't understand how it works and I want to understand. Heck, I don't understand how our brains are able to do it either, but I don't really know anything about neurology beyond the basic facts most people know. I do know the basics of machine learning well enough that I'm hoping I can get a high level understanding of the parts of the math behind speech recognition that distinguish between speaker specific and content specific information. I can't think of a particular reason I'm curious about it -- I just am. $\endgroup$
    – Mikayla Eckel Cifrese
    Commented Jul 6, 2023 at 5:33

2 Answers 2

Reset to default
2
$\begingroup$

I would slightly disagree with Ryan's answer: the fundamental frequency is mainly specific to a speaker. Sounds are defined by other frequency patterns.

Vowels, for example, have two bands of energy in different frequency ranges, called formants. Depending on the speaker, these will be in different actual frequency ranges, but the general pattern will remain the same. They are linked to the fundamental frequency F0 in that it is multiples of F0 which can be seen in a spectrogram.

There is obviously lot of variation in the exact frequencies involved because of circumstances. Apart from different speakers with different physiology, having a cold also changes your frequency output. For speech recognition you need various mechanisms to map a sequence of frequency values to phonemes and then words. When I was studying phonetics in the 1990s, the upcoming tool was Hidden Markov Models, and neural networks were being explored as well, but were still in their infancy.

I'm not sure what currently is used for speech recognition, as I haven't done any phonetics for years, but I would expect NNs to be more prevalent these days.

To summarise: there are some features in the frequency pattern of an utterance which give you an indication of the (language) sounds produced, whereas others reflect the physiology of the vocal tract (and are thus speaker-specific). But they are not unique, as there is no "voice print", which was some time ago touted as a finger print equivalent.

Improve this answer
$\endgroup$
1
$\begingroup$

A sound byte can be decomposed into a set of features which are distinguishable by a classifier. The most important feature extracted is the fundamental frequency. This is the lowest frequency function from approximating the windowed waveform with Fourier series. It is fairly unique for each phoneme. In addition, each phoneme can be represented as a series of "frequency bins" where the intensity is the height of each bin. These features are not only unique to each phoneme, but also to each speaker. Additionally, statical moments can be calculated for each window and used as features.

A Recurrent Neural Network (there are many techniques) can be used to agglomerate features into a recognition model. The idea is to learn the probability of one set of features following another set in time.

For an example, a 20 second sound byte of a speaker can be broken into 20 1-second windows of speech. Each window represents a phoneme or some part of speech (depending on window-size, speaker speed, etc). A Fourier transform is applied on a window to derive a feature set. The feature set is fed into an RNN to predict the speaker (or text-to-speech, TTS). There are many inter-related applications here.

Hope this was helpful.

Improve this answer
$\endgroup$
5
  • $\begingroup$ "A sound byte can be decomposed into a set of features which are distinguishable by a classifier." What If like to know is, which classifiers are relevant to distinguishing between different speakers and which are relevant to distinguishing between different messages? e.g. If had a recording of a person with a British accent accent saying the word "water" and another recording of a person with an American accent saying, which features of the data would recognize the different accents and which ones would "recognize" (for lack of a better term) that it's the same word being spoken? $\endgroup$
    – Mikayla Eckel Cifrese
    Commented Jul 6, 2023 at 20:01
  • 1
    $\begingroup$ Typically there are (at least) two stages. The first stage picks up features like phonemes and syllables. The second stage identifies specific words and picks up things like which homophone was meant (e.g. they're or their). $\endgroup$
    – Bruce Adams
    Commented Jul 6, 2023 at 23:21
  • $\begingroup$ @MikaylaEckelCifrese the same models can be used for both. RNNs, LSTMs, transformers, etc can all be used for either task. The difference is in the target for training which comes down to calculating loss for updating the network. Is the network "output" a vector that represents a speaker or a word. This is training dependent. I would recommend becoming more familiar with ML in general before tackling this specific task. $\endgroup$
    – Ryan
    Commented Jul 7, 2023 at 16:41
  • $\begingroup$ @Ryan, but about when the software needs to recognize both speaker specific and content specific information? Also, I don't currently have any intention of trying to make a speech recognition system, as that's definitely beyond my current skill level -- I'm just curious about how such software works at a high level. $\endgroup$
    – Mikayla Eckel Cifrese
    Commented Jul 7, 2023 at 19:16
  • $\begingroup$ @MikaylaEckelCifrese, Many features can be re-used for both tasks, however most models in practice will have common feature extraction layers and then have models that perform specific tasks. There might be several layers on top of the base extraction for one task, and separate layers for another task. The software has multiple models with some common layers. $\endgroup$
    – Ryan
    Commented Jul 12, 2023 at 20:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .