0
$\begingroup$

We want to try and distinguish real voices from (deep)fake voices using the graphs generated by a discrete fourier transform (generated from .wav audio files). We know from each image if it is a real or a fake voice, so it's a supervised classification problem. An image would look like this:

enter image description here

We think that real voices generate a graph with clear spikes, whereas fake voices have more noise resulting in less clear spikes. For this reason, we thought of using a CNN to take such an image as input (with x and y-axes ommited), and classify it as real or fake. Our concern is that it's actually a graph and not an image of an object, so we're not sure if this would be a good approach. We could also use the arrays generated from the fourier transform, but we're not sure how we could use that as input as we want to classify if it's real or fake, and not predict y for each x.

$\endgroup$
0
0
$\begingroup$

There no problem with the use of the data in form of an array to classify, whether the audio belongs to a real or fake voice. Just use 1d convolutional neural network with downsamplings or some global pooling operations, such that in the final layer the temporal extent of the signal has length 1. This would be the logit for binary classification.

However, as far as I understand, you get rid of phase after the Fourier transform, but it can be useful for the prediction. Probably, a better approach would be to use mel_spectrogram https://en.wikipedia.org/wiki/Mel-frequency_cepstrum for this problem.

$\endgroup$
1
  • $\begingroup$ Thanks! Wouldn't using a 1D CNN require time-series data? As you said, at this point we only have the frequencies vs. amplitude left. We're aware that using mel spectrograms could(possibly) perform better in such a CNN, but we were wondering if other methods could also work while making sense (so if it makes sense to use graph-like images for a CNN, or if there's some factor(s) as to why not) $\endgroup$ Jun 9 at 11:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.