4
$\begingroup$

I want to program and train a voice cloner, in part to learn about this area of AI, and in part to use as a prototype of audio for testing and getting feedback from early adopters before recording in a studio with voice actors. For the prototype, I have a set of recordings from voice actors. I would like to record my voice, in English or other languages, then run a neural network and produce an audio with the same text, intonation and emotion but with roughly the actors' voices. It doesn't need to be perfect; 80% right and believable would be enough to get good feedback and reach a final version of the script before recording. I have 30 minutes to one hour of utterances from each voice I want to clone.

The closest I have found is Resemble.ai, which has an impressive video, but the public plan is only in English and other languages are prohibitively expensive. The engineer published a masters' thesis as an open-source project, but this project does only text-to-speech, not speech-to-speech. Another startup is play.ht, but again it seems to be English-only.

This open source project seems to do what I want, cloning Kate Winslet's voice, but it has no installation instructions and so I haven't tried yet.

Can you recommend an open-source project, ideally in Python and Tensorflow, to roughly replace a voice with another?

Note: This question is similar to What is the State-of-the-Art open source Voice Cloning tool right now? , except that that question is old and the project mentioned only does text-to-speech, not speech-to-speech.

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ Interesting, thanks for the links. Wouldn't it be possible though to have speech-to-text nn and then text-to-speech + specific voice? $\endgroup$
    – user57690
    Commented Mar 14, 2023 at 9:01
  • $\begingroup$ @MahNeh A text-to-speech would remove most of the artistic effects from tone of voice, such as pauses, emphasis, cadence, irony, etc. We tried producing video with text-to-speech in standard voices and our users did not like it at all, so we need something with a more human touch. $\endgroup$
    – emonigma
    Commented Mar 14, 2023 at 11:16
  • 1
    $\begingroup$ You are correct. FYI, I searched for updated forks of the project you name (deep voice conversion), but last updates are from 2019 in the most popular forks (13 stars, 3 stars etc.) I would not go for that. $\endgroup$
    – user57690
    Commented Mar 14, 2023 at 11:53
  • $\begingroup$ @MahNeh Yes, I saw the forks, but as you say, they seem outdated. It may be a sign that I need to do a fork myself and keep it current! $\endgroup$
    – emonigma
    Commented Mar 14, 2023 at 12:18
  • 1
    $\begingroup$ Yes, not a bad idea, you may check the last 20-50 open and closed issues, and try to update it. Would this change TF1 to TF2 ? Side note: I tried a yolo version a while ago and it was impossible to fix lol (looking at the issues would have helped.) $\endgroup$
    – user57690
    Commented Mar 14, 2023 at 12:50

2 Answers 2

Reset to default
2
+100
$\begingroup$

Additional projects that might be of interest:

A neural voice cloning system is introduced, using a few audio samples to create personalized speech interfaces. Two approaches are explored: speaker adaptation, which fine-tunes a multi-speaker model with cloning samples, and speaker encoding, which trains a separate model to infer new speaker embeddings from cloning audios. Both methods achieve good performance in terms of speech naturalness and similarity to the original speaker. Although speaker adaptation offers better naturalness and similarity, speaker encoding demands less cloning time and memory, making it suitable for low-resource deployment.

Here is an open-source implementation of the paper, but the GitHub page says the project is archived since February 2021 and read-only.

In this paper, the authors present a novel one-shot voice cloning algorithm called Unet-TTS that has good generalization ability for unseen speakers and styles. Based on a skip-connected U-net structure, the new model can efficiently discover speaker-level and utterance-level spectral feature details from the reference audio, enabling accurate inference of complex acoustic characteristics as well as imitation of speaking styles into the synthetic speech. According to both subjective and objective evaluations of similarity, the new model outperforms both speaker embedding and unsupervised style modeling (GST) approaches on an unseen emotional corpus.

The GitHub repo has a link to a Google Colab.

(Not Open Source, but has a free tier. Voice cloning becomes available in the Starter Tier, starting at 5$/month.)

ElevenLabs initially built new text-to-speech models which rely on high compression and context understanding to render human speech ultra-realistically. Their tools aim to provide the necessary quality for voicing news, newsletters, books and videos. They also offer a suite of tools for voice cloning and designing synthetic voices.

(Not Open Source, but has a free tier and is a partner of the Open Voice Network, a non-profit industry association dedicated to making voice technology worthy of user trust and it operates as a directed fund of The Linux Foundation.)

Voice cloning is part of the enterprise plan with custom pricing and requires 2-8 hours of recorded utterances following their script. See an example of original and cloned voice in English on YouTube. Although it sources non-English voices from partners such as Google and Amazon, it does not seem to support voice cloning in languages other than English.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks! I didn't try the first, Neural Voice Cloning with a Few Samples. I tried the second, Unet-TTS, out-of-the-box and it fails with ERROR: Could not find a version that satisfies the requirement tensorflow-gpu==2.6.0. I tested ElevenLabs and liked the results very much, but unfortunately they don't have my target language. BeyondWords seems very good, but they require 2-8 hours of audio, which is infeasible in my case. So, if my use case had the luck of falling into ElevenLabs' language, I would use that, and so I'm accepting the answer. $\endgroup$
    – emonigma
    Commented Nov 14, 2023 at 19:43
3
$\begingroup$

Tensorflow code for "one-to-one" style transfer:

https://github.com/phiana/speech-style-transfer-vae-gan-tensorflow

it's the implementation of a 2021 paper. Speech style transfer, voice cloning or speech-to-speech synthesis are the keywords. Further research (looking at the state of the art) would yield some papers:

It seems to be a research area without much activity though. Maybe you could add something interesting :)

Improve this answer
$\endgroup$
1
  • $\begingroup$ I tried to install the VAE-GAN out-of-the-box and it failed; also, for my purpose I need a many-to-many style transfer model. The MIST-Tacotron paper seems interesting but I couldn't find code or a repo. Expressive Neural Voice Cloning is interesting because it has both text-to-speech and speech-to-speech in the target voice. $\endgroup$
    – emonigma
    Commented Nov 14, 2023 at 19:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .