What would be the best approach to teach an AI to learn how to "sing" along a beat?

Question

I have heard and read about HyperGAN, LSTM and a few other techniques, but I have a hard time piecing the overall concept together.

End Goal

Being able to input an instrumental and get an output of how to sing to that instrumental.

My Dataset

I have extracted pitch points from thousands of actual acapellas from real songs.

My Theory

Feed the AI a pitch point PLUS say 19 thousand points of the original song instrumental.

Illustration

The red line (on top) is the pitch viewed vertically (lower pitch down, higher pitch up) of the voice sung by the singer over time viewed horizontally.

The bottom image is the song's frequency viewed vertically (lower freq down, higher freq up) viewed horizontally over time.

We take a point in time of the instrumental, say 0 minutes 30 seconds, and extract 19k points of the FFT spectrum vertically and call this a frame.

We also take the same point in time of the voice pitch, and also refer to this as a frame.

So now we have a frame which contains 20 thousand data points, one being the pitch of the voice, and the rest being the frequencies of the songs content.

QUESTION

What kind of model could be used to teach the AI the correlation of the voice and the instrumental?

And also, I have a hard time understand how, once the AI is trained, how could just an instrumental be fed to the AI to output pitch values of how one COULD sing along to the song.

Like, training we need to input 20 thousand values, but when we want the AI to sing for us using just an instrumental, would it not still expect voice pitch input? At what layer would the instrumental be tapped into? At the outer most right layer?

EDIT

My mind has been working on this in the background throughout the day, and I am wondering if instead of feeding 19k points of instrumental data each frame (which would be points from the frequency domain), one could just feed the instrumental frame points (which would be points from the time domain).

Maybe that would be better, but then maybe the AI would get less "resolution" to work with, but could be trained faster (less computing power needed).

Let's say the frequency domain is fed (higher resolution), the AI could potentially find correlations from low notes, mid notes and high notes, in any combination (more computing power needed).

Answer 1

You have to define a function between voice and a corresponding sound, but there is no a best way to do it, it depends on what kind od output you care, for example you could create an algorithm that receives music and siging as separate "files" and then that algorithm finds the relation between them now you can give that algorithm a new music only file and it will create the singing part with the function of the previous music-singing "system". This is sing this music as the other one was sang.

At least that is the way i would do it.