# How is speech recognition software able to distinguish between different speakers and yet still understand them all?

Some background: I'm an EE major and data science minor, so I have a basic understanding of machine learning - I had a one semester course on it where we covered some of the most commonly used algorithms such as Naive Bayes' and K-nearest-neighbors, as well as the basics of how artificial neural networks work. I've also had EE classes on signal processing, so I'm familiar with the Fourier transform and the like. So I have a general idea of how different algorithms can, for lack of a better term, "pick up on" patterns in the training data and make predictions on the testing data, which will hopefully be reasonably accurate. And I have a general idea of how signals can be represented as discrete binary sequences which could then be used to train an ML model.

What I want to know is, what sorts of patterns, in the mathematical sense, in audio recordings of speech specifically, do these algorithms find? e.g., What allows the Alexa app to both recognize when I'm speaking vs when it's my grandma speaking, and yet still understand the same commands from both of us? What, mathematically, makes, e.g., a recording of me saying "Alexa play music" both distinguishable from a recording of a different person saying the same command, and yet still recognizably contain the same information of the command? In other words, how do the algorithms extract and match the speaker-specific and content-specific information from the speech signals?

• Is there a reason you find this odd? After all, you understand yourself, your grandmother, etc.
– Dave
Commented Jul 5, 2023 at 22:18
• @Dave, it's not that I find it odd -- it's just that I don't understand how it works and I want to understand. Heck, I don't understand how our brains are able to do it either, but I don't really know anything about neurology beyond the basic facts most people know. I do know the basics of machine learning well enough that I'm hoping I can get a high level understanding of the parts of the math behind speech recognition that distinguish between speaker specific and content specific information. I can't think of a particular reason I'm curious about it -- I just am. Commented Jul 6, 2023 at 5:33

I would slightly disagree with Ryan's answer: the fundamental frequency is mainly specific to a speaker. Sounds are defined by other frequency patterns.

Vowels, for example, have two bands of energy in different frequency ranges, called formants. Depending on the speaker, these will be in different actual frequency ranges, but the general pattern will remain the same. They are linked to the fundamental frequency F0 in that it is multiples of F0 which can be seen in a spectrogram.

There is obviously lot of variation in the exact frequencies involved because of circumstances. Apart from different speakers with different physiology, having a cold also changes your frequency output. For speech recognition you need various mechanisms to map a sequence of frequency values to phonemes and then words. When I was studying phonetics in the 1990s, the upcoming tool was Hidden Markov Models, and neural networks were being explored as well, but were still in their infancy.

I'm not sure what currently is used for speech recognition, as I haven't done any phonetics for years, but I would expect NNs to be more prevalent these days.

To summarise: there are some features in the frequency pattern of an utterance which give you an indication of the (language) sounds produced, whereas others reflect the physiology of the vocal tract (and are thus speaker-specific). But they are not unique, as there is no "voice print", which was some time ago touted as a finger print equivalent.

A sound byte can be decomposed into a set of features which are distinguishable by a classifier. The most important feature extracted is the fundamental frequency. This is the lowest frequency function from approximating the windowed waveform with Fourier series. It is fairly unique for each phoneme. In addition, each phoneme can be represented as a series of "frequency bins" where the intensity is the height of each bin. These features are not only unique to each phoneme, but also to each speaker. Additionally, statical moments can be calculated for each window and used as features.

A Recurrent Neural Network (there are many techniques) can be used to agglomerate features into a recognition model. The idea is to learn the probability of one set of features following another set in time.

For an example, a 20 second sound byte of a speaker can be broken into 20 1-second windows of speech. Each window represents a phoneme or some part of speech (depending on window-size, speaker speed, etc). A Fourier transform is applied on a window to derive a feature set. The feature set is fed into an RNN to predict the speaker (or text-to-speech, TTS). There are many inter-related applications here.