I am working on a research project to use (demonstrate) the possibilities of Machine Learning and AI in artistic projects. One thing we are exploring is demonstrating deep fakes on stage. Of course, a deep fake is not easy to make. Still, we are exploring creating a "minor quality" deep fake live on a stage (or maybe in some other ways where people can make deep fakes of themselves) in which we put words into someone's mouth. I discovered that a semi-nice deep fake of the facial movement is possible, now a also want to add voice.
There are a lot of text-to-speech systems, which allow using a voice that is created from the recording of the voice of a real person. That is already nice.
The video is based on the facial movements of another person. So the Audio has to match the facial movements. The easiest way would be, if the "fake" voice says it exactly the same way, as the person doing the "facial acting" for the deep fake said it.
Is it possible to do a fake voice of a person in this way:
- Another person (the actor, or source) speaks the words in his voice and records it.
- The person with the faked voice (the destination) gives a voice sample with some random spoken text.
- An AI/algorithm/whatever modifies the recording from (1) in such a way, that the tone/voice is modified so it matches the voice from (2).
Do systems/research like this exist? I did not find anything using google, but maybe I did not use the correct keywords.