1
$\begingroup$

Background:

I am working on a research project to use (demonstrate) the possibilities of Machine Learning and AI in artistic projects. One thing we are exploring is demonstrating deep fakes on stage. Of course, a deep fake is not easy to make. Still, we are exploring creating a "minor quality" deep fake live on a stage (or maybe in some other ways where people can make deep fakes of themselves) in which we put words into someone's mouth. I discovered that a semi-nice deep fake of the facial movement is possible, now a also want to add voice.

There are a lot of text-to-speech systems, which allow using a voice that is created from the recording of the voice of a real person. That is already nice.

The video is based on the facial movements of another person. So the Audio has to match the facial movements. The easiest way would be, if the "fake" voice says it exactly the same way, as the person doing the "facial acting" for the deep fake said it.

Question:

Is it possible to do a fake voice of a person in this way:

  1. Another person (the actor, or source) speaks the words in his voice and records it.
  2. The person with the faked voice (the destination) gives a voice sample with some random spoken text.
  3. An AI/algorithm/whatever modifies the recording from (1) in such a way, that the tone/voice is modified so it matches the voice from (2).

Do systems/research like this exist? I did not find anything using google, but maybe I did not use the correct keywords.

Improve this question
$\endgroup$
4
  • $\begingroup$ I think this is probably possible. The research I'm aware of is about images. Have a look at CycleGAN, which learns to convert horses into zebras, with just a bunch of horse pictures and an unrelated bunch of zebra pictures. $\endgroup$
    – user253751
    Sep 18, 2021 at 13:39
  • $\begingroup$ also StyleGAN which transfers the "artistic style" from one picture onto another picture. Voice is just "artistic style" but for sound. $\endgroup$
    – user253751
    Sep 18, 2021 at 13:40
  • $\begingroup$ of course there is no way to know for sure without actually trying it and probably trying a bunch of dead ends before it might actually work $\endgroup$
    – user253751
    Sep 18, 2021 at 13:41
  • 1
    $\begingroup$ @user253751 Thanks for your input! The repos you are listing are also interesting. Unfortunately this is out of my skill and time possibilities. $\endgroup$
    – Nathan
    Sep 18, 2021 at 14:07

0

Reset to default

You must log in to answer this question.